[prometheus-adapter] Alerts when prometheus-adapter fails to fetch metrics.

[kmdkuk]

What

Since there is no way to expose the metrics in the prometheus-adapter itself, there is currently no way to detect if the prometheus-adapter is failing.
If prometheus-adapter does not work, HPA using custom metrics does not work well.

How

• Consider a contribution that exposes metrics to prometheus-adapter.
• Expose metrics from log by promtail functionality.
  • {namespace="prometheus-adapter"} |~ ". *unable to fetch metrics.*" to check the error log.
    • https://github.com/kubernetes-sigs/prometheus-adapter/blob/v0.9.1/pkg/custom-provider/provider.go#L261
    • https://github.com/kubernetes-sigs/prometheus-adapter/blob/v0.9.1/pkg/resourceprovider/provider.go#L163
    • https://github.com/kubernetes-sigs/prometheus-adapter/blob/v0.9.1/pkg/external-provider/provider.go#L57
• Verify that the query used by prometheus-adapter does not fail.
  • At the moment {__name__=~"^container_. *",container!="POD",namespace!="",pod!=""} is failing.
  • https://github.com/cybozu-private/neco-apps/blob/d1a01cce25b36e6c56f6d3ac54de355520a0e185/argocd-config/base/prometheus-adapter.yaml#L58

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions