Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change alert config to have a rule for each alerting window #82

Open
IvanMerrill opened this issue Aug 9, 2023 · 0 comments
Open

Change alert config to have a rule for each alerting window #82

IvanMerrill opened this issue Aug 9, 2023 · 0 comments

Comments

@IvanMerrill
Copy link

IvanMerrill commented Aug 9, 2023

Currently, the alerting rules are just the default ones created by Sloth. Sloth creates one alert rule for the paging severity and one for the ticket severity. However in the paging severity there are two rules evaluated with a big OR in between. Here's an example:

     (
          max(slo:sli_error:ratio_rate5m{sloth_id="autometrics-success-rate-95", sloth_service="autometrics", sloth_slo="success-rate-95"} > (14.4 * 0.05)) without (sloth_window)
          and
          max(slo:sli_error:ratio_rate1h{sloth_id="autometrics-success-rate-95", sloth_service="autometrics", sloth_slo="success-rate-95"} > (14.4 * 0.05)) without (sloth_window)
      )
      or
      (
          max(slo:sli_error:ratio_rate30m{sloth_id="autometrics-success-rate-95", sloth_service="autometrics", sloth_slo="success-rate-95"} > (6 * 0.05)) without (sloth_window)
          and
          max(slo:sli_error:ratio_rate6h{sloth_id="autometrics-success-rate-95", sloth_service="autometrics", sloth_slo="success-rate-95"} > (6 * 0.05)) without (sloth_window)
      )

When this alert triggers you cannot see if it's triggered because of the 1h/5m time window + burn rate rule, or the 6h/30m rule. This information also isn't included in any label or anything. It could be worth breaking this rule out into two different rules enabling the user to understand the time frame and burn rate involved in generating this alert. We could include this information in a label on the alert as well to allow it to be better understood and displayed in explorer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant