Promxy uses /api/v1/series for ALERTS_FOR_STATE

[perederyaev]

Hi!

Looks like restoring alert's state feature dosn't work.
Promxy uses /api/v1/series for ALERTS_FOR_STATE it can't restore alert's state because result from /api/v1/series contains no timestamp.

Tested on most recent versions and current master with prometheus and victoriametrics as backends.

[jacksontj]

Thanks for opening an issue! I spent some time attempting to reproduce the error you are describing but I am unable to do so. When I run locally using the demo_robust.conf configuration and the rules.alertbackfill CLI flag it works fine locally (I see the query happening; and the alert goes to red within ~5s of startup.

I did some digging and I also can't see how a backfill query would hit the /series endpoint either. If we look at the backfill code we can see that it is doing a QueryRange which doesn't get converted to anything besides a query range all the way down.

So at this point I am unable to reproduce your reported issue; if you could provide more detail so that I can reproduce the issue; or some additional logs/errors/etc. that would go a long way towards getting a repro case.

[perederyaev]

Hi, the issue is occurring when rules.alertbackfill option is not specified.
As I understand, rules.alertbackfill helps when no ALERTS_FOR_STATE available, but I have correct ALERTS_FOR_STATE metric and just want standard prometheus alert's restoration feature to work.

[jacksontj]

I have correct ALERTS_FOR_STATE metric and just want standard prometheus alert's restoration feature to work.

AH; that makes a lot more sense :)

So I don't believe the issue is anything to do with the return value -- but everything to do with the query. Prometheus' restore query is VERY particular -- and it expects the labels to be exactly as it would generate them. If you enable debug logging you can actually see the exact query that it will construct -- and it will expect the results to have the exact same labels (no more, no less). So if you have any labels defines on the ServerGroup -- this will generally "break" that query mechanism.

IMO prometheus' query is being too strict -- but thats what it does. So I'd suggest checking if that is the case -- and seeing if you can adjust your configuration to make that align. If not; I am definitely open to suggestons on how to best make that work (I don't think I know any users that rely on the ALERTS_FOR_STATE default backfill -- as its pretty finnicky because of the exactness of that match requirement.).

[jacksontj]

As its been a few months with no response I'm going to close this out (assuming it is abandoned). If this is still an issue please feel free to re-open!