-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
promxy did not get external_labels from server_groups #652
Comments
First off, thanks for reaching out! I am a bit confused by the report/question here. Some of this is likely due to
This means that these labels aren't returned in the metrics queries -- but rather when being sent out to external systems (e.g. alertmanager). This is especially murky since it indicates that it shows up in "remote storage" but not in the regular query interface (super confusing). Promxy is passing the external labels config to the prometheus rule manager so these external labels are added to alerts fired to alertmanager; but that is the only current usage of these external labels. In your usage; where are you expecting to see these external labels? |
Hi @jacksontj
If Prom A has config
and Prom B has config
and Prometheus (frontend) has config
and both of them have metric
Next step - let's replace Prometheus Frontend with Promxy, with following setup:
Then similar request for metric1 to promxy would return
(and it would be metric1 from PromA btw). So, if you know external labels for PromA and PromB fix is obvious:
It's obviously restoring old behavior. So, it's bad for breaking compatibility (but IIRC promxy do not promise any) and I can imagine some issues only if external labels is dynamic and not easily obtainable in promxy - which is kinda theoretical. |
That's because your frontend uses Federation. So that's native Prometheus feature. Promxy doesn't act as Federator, so it has no way to get the Prometheus external_labels as they're not stored anywhere in TSDB but rather added dynamically on federation/remote_write/alertmanager. |
Hi @Giovanniuum, |
This part I guess I'm a bit confused on. If I take a look at stock prometheus promxy's behavior seems consistent. As an example; if we use demo.robustperception.io as our example: the config has the following:
But when I do a query for metrics I get back:
Note that the The use-case I've seen most for So unless I've missed something, this feels like its working as intended? (even though the intent is a bit confusing :D ). |
Hi @jacksontj
In simple query it does not, indeed. You need 2 prometheus instances, demo installation do not have it.
Yes. Or proxy. Prometheus also can be proxy to other prometheuses (without data merging ofc) and I faced issue when replaced proxy prometheus with promxy. I changed description above to reflect it.
This issue is not about sending external labels (I don't know does promxy exen support it) but receiving external labels from downstream prometheuses. In that aspect promxy breaks compatibility with stock prometheus.
I'm not challenging intention of the project, indeed. If it feels right and promxy should do exactly that - that's more than fine. I just noted that in this aspect promxy is not compatible with stock prometheus - which exactly "derive and merge what downstreams have configured" - and this behaviour should be or corrected or highlighted in readme, or at least in this issue. Feel free to close it, it would server its purpose even if closed. |
Oh; I am not aware of this configuration for prometheus. If you have some links to docs/configs maybe I can read up on that and come up with a better solution here.
To be a bit pedantic here; the issue (at least as I'm seeing it) is that the downstream prometheus instances aren't sending these external labels. So this isn't a case of promxy eating the labels -- but rather that these
I would love some more context on this prometheus-as-a-proxy stuff; since I am completely unaware of that behavior in prometheus (and unable to find anything in my little bit of googling). Fundamentally promxy is just hitting the query API the same way a user would if you went to the prometheus HTTP UI. So if the labels aren't there (at least how it is today) then they won't show up. If prometheus has some mechanism to conditionally add those in the response -- I'd be VERY curious what that is (since I might be able to encorporate that). |
Given where this left off it sounds like we can close this out. If there is more context from yourself or someone else on this "prometheus as a proxy" I would be very interested in understanding how that fits into the picture. Otherwise it sounds like we should close this out for now (if you disagree definitely feel free to re-open :) ) |
Sure, no problem. |
Promxy do not retrieves any "external_labels" from server_groups, which breaks compatibility with Prometheus.
In our case we migrated to labels, but I can imagine use case where it can be tricky.
The text was updated successfully, but these errors were encountered: