Reset scrape_errors_total metric

[keithf4]

Describe the bug
Been trying to narrow this down to a repeatable use-case, but having a hard time. Sometimes when a metric has a scrape error, after fixing it so the error is no longer occurring, the scrape_errors_total metric does not clear automatically. But other times it does. The only way I've found to consistently fix this is to restart the exporter but this is not ideal.

To Reproduce
Steps to reproduce the behavior:

Unable to reproduce consistently. But just do something that causes scrape_errors_total to have values, fix that problem, and see if it goes away without restarting the exporter.

Expected behavior
Since we monitor scrape_errors_total to see if there are any scrape errors occurring, it would be good if this metric could clear itself consistently without restarting the exporter so we know the issue has been resolved.

[burningalchemist]

Hey @keithf4,

scrape_errors_total is a counter metric which will only increase.

On the Prometheus side, I'd say using increase() function (or rate()) within a time window should provide enough visibility (similar to how we deal with http_errors_total metric, for instance). It should not matter whether you restart sql_exporter or not - Prometheus should keep track of that.

You've mentioned however that sometimes the metric clears automatically - I'm more concerned about it as it shouldn't be the case (unless you restart the app). I'd like to get more details on it as consistency is the goal here.

So far I'm not into managing metrics (it's one more goroutine to keep around), in the end we only export data.

[keithf4]

Hmm ok. I'll try using the increase(). The inconsistency with it resetting is what had me confused as to how to use it.

However, I'd still suggest further that it reset automatically somehow. Each individual metric failing causes scrape_errors_total to have another metric as well. Which is good so you know which metrics are failing. However say something happens and you have 1000+ metrics fail. You now have an additional 1000 failure count metrics that will stick around and be pulled into Prometheus until the exporter is restarted.

[burningalchemist]

Fair enough. 👍

Let me bring it to the list of ideas, since it's not a bug issue (rather an improvement)?
I'll think on what I can do, and try it out first.

[burningalchemist]

Hey @keithf4, I've created a branch with the feature implemented. The timer is controlled by the scrape_error_drop_interval parameter (set to 0 by default i.e. disabled). Setting it to 5m or 24h will make sure the scrape_errors_total is dropped at the right time.

Feel free to try it out if you have time/will, and available environments.

I'll let it settle for a while before opening a PR, but overall it seems working fine, and might enable your scenarios (especially, since you mentioned potential 1000 failure count metrics 😆).

[keithf4]

Don't have an easy way to test a branch at the moment and our build team is a bit swamp. If this works as described, though, this would definitely solve the issue. Thanks so much for the prompt responses!

[burningalchemist]

No worries, I'll test it on my own in the upcoming days, and hopefully release a new version at the beginning of the next month. 👍