Prometheus errors in OpenShift with default configuration

[lgoral]

I'm using version 0.58.0 with configuration

clusterName: xxxx
  splunkPlatform:
    token: xxxxx
    endpoint: xxxx
    logsEnabled: true
    index: openshift
    metricsEnabled: true
    metricsIndex: openshift_metrics

  logsEngine: otel
  distribution: openshift

and on each splunk agent "splunk-collector-agent-*" have errors

2022-09-23T04:23:42.538Z error prometheusexporter/prometheus.go:141 Could not get prometheus metrics {"kind": "receiver", "name": "receiver_creator", "pipeline": "metrics", "name": "smartagent/kubernetes-proxy/receiver_creator{endpoint=\"10.210.0.9\"}", "monitorType": "kubernetes-proxy", "error": "Get \"http://10.210.0.9:29101/metrics\": dial tcp 10.210.0.9:29101: connect: connection refused"} 2022-09-23T04:23:48.249Z info fileconsumer/file.go:159 Started watching file {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/openshift-marketplace_redhat-marketplace-lm2w9_91649261-f4be-4e16-9b10-fb7c5c038084/registry-server/0.log"}

openshift version: 4.10.26

[atoulme]

You can look here to understand what is happening: https://github.com/signalfx/splunk-otel-collector-chart/blob/f008069bfed00feb1286e9c85c1d024eb1913fe2/helm-charts/splunk-otel-collector/templates/config/_otel-agent.tpl
We are trying to scrape kube-proxy metrics, but they're not exposed in your cluster.

You can disable the proxy metrics collection by setting agent.controlPlaneMetrics.proxy.enabled to false.
You can also configure kube-proxy to expose metrics.