Problems with Prometheus on OTel 1.9

[dominikjeske]

Package

OpenTelemetry

Package Version

Package Name	Version
OpenTelemetry.Api	1.9.0
OpenTelemetry	1.9.0
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.9.0
OpenTelemetry.Exporter.Prometheus.AspNetCore	1.9.0-beta.2
OpenTelemetry.Exporter.Prometheus.HttpListener	1.9.0-beta.2

Runtime Version

net8.0

Description

Today I tried to use OTel 1.9 but still I have problems with metrics. Prometheus logs:

prometheus[603702]: ts=2024-06-26T13:18:31.847Z caller=scrape.go:1607 level=warn component="scrape manager" scrape_pool=webapi target=
http://xxxxxxxxx:8080/metrics
msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

When rollback to 1.7 everything looks good. Same problem was on 1.8 #5613 and issue was closed as resolved.

Is this a bug or we have to use new prometheus to handle metrics from 1.8+?

Steps to Reproduce

Just use standard OTel and serve prometheus endpoint

Expected Result

Prometheus agent is grabbing data with success

Actual Result

Errors from description

Additional Context

No response

[Kielek]

@dominikjeske, could you please provide Minimal, Reproducible Example? If you provide code which we can easily reproduce it should reduce time to solve.

If you find the root cause of the issue/find the way how to solve it will be great if you can share. PR is more than welcomed.

BTW Prometheus is supporting OTLP now. This might be an option to you to solve it (technically workaround, but it is the recommended protocol).