query(fe):/api/v1/label/__name__/values produces empty result set with prometheus 3.0

[rsommer]

Thanos, Prometheus and Golang version used:
thanos 0.36.1 (upstream docker image)
prometheus 3.0.0 (upstream docker image)

Object Storage Provider:
Ceph

What happened:
After updating to prometheus 3.0, the query for metric names used in grafana stopped working while using thanos' query frontend. The query to /api/v1/label/__name__/values?start=1731918720&end=1731922380 produces

{"status":"success","data":[]}

Issueing the same query directly against prometheus 3.0 works fine. Switching prometheus back to 2.55.1, the query works again via thanos' query frontend.

What you expected to happen:
I expected to get the same results when querying via thanos' query frontend

How to reproduce it (as minimally and precisely as possible):
If you have a working setup with thanos and prometheus < 3.0, issue
curl "http://thanosqueryfrontend:9090/api/v1/label/__name__/values?start=1731918720&end=1731922380"
with adjusted time ranges - you'll get an answer with the data field containing the available metrics.
Update prometheus to 3.0 and use the same query again, you'll get
{"status":"success","data":[]} with an empty data field.

Environment:

• OS: Debian GNU/Linux 12 (bookworm)
• Kernel: Debian 6.1.76-1

[dosubot]

Hello @rsommer! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve this issue.

The issue you're experiencing with the Thanos query frontend returning an empty result set when using Prometheus 3.0 could be related to several factors. Here are some potential causes and considerations:

1. Invalid Timestamps: Ensure that the start and end parameters are valid timestamps. If they cannot be parsed correctly, it could lead to an empty result set ^[1].

2. End Before Start: Verify that the end timestamp is not before the start timestamp, as this would result in an error and no data being returned ^[1].

3. API Changes: There might be changes in the API responses or behavior between Prometheus 2.x and 3.0 that affect how Thanos processes the requests. Although specific changes weren't found in the repository, this is a common area to investigate when upgrading major versions.

4. Compatibility Issues: There might be compatibility issues between Thanos 0.36.1 and Prometheus 3.0 that are not documented in the repository. Testing with different versions or checking for any available patches or updates might help.

5. Configuration Adjustments: Consider reviewing any configuration settings in Thanos that might need adjustment when upgrading Prometheus to 3.0, although specific configurations weren't identified in the repository.

Since the query works directly against Prometheus 3.0 but not through Thanos, it suggests that the issue might be specific to how Thanos interacts with the new Prometheus version. You might want to check for any error logs or debug information in Thanos that could provide more insights into the issue.

To continue talking to Dosu, mention @dosu.