Crons: Detect ingestion outages during clock ticks

[evanpurkhiser]

There is a failure scenario that can be very disruptive for our customers.

If we have an outage in our ingestion of cron check-ins, specifically where we are dropping check-ins, then we may incorrectly mark customers cron monitors as having missed check-ins. This only happens if we drop check-ins, if we are delayed with check-ins, the clock ticks which drive missed and time-out detections will slow down to match the consumption of check-ins in our topic.

This is highly problematic as it means customers are unable to trust that cron alerts are accurate. This is, however, a difficult problem, since if check-ins never make it into the check-ins topic, how can we differentiate between a customers job failing and not sending a check-in, and us failing to ingest their check-in?

In most of our ingestion failure scenarios we have had a significant drop in check ins. That typically looks something like this:

Improved behavior

If we were able to detect this extreme drop in volume, we could create clock ticks that are marked as being unknown ticks. meaning we are moving the clock forward, but we have a high certainty that we may have lost many check-ins. When this happens instead of creating missed and timed-out check-ins that trigger alerts, we can create missed check-ins that have a "unknown" status and mark in-progress check-ins that are past their latest check-in time as "unknown", again not alerting customers. Once we are certain that we have recovered from our incident, the clock will resume producing ticks that are regular, and not marked as unknown.

Detecting ingestion volume drops

The tricky part here is deciding if we are in an incident state. Ideally we are not relying on an external service telling us that we may be in an incident, since that service itself may be part of the incident (eg, if we had relay report that it was having problems, there's no guarantee that when it's having problems it will just fail to report to us).

My proposed detection solution is rather simple. As we consume check-ins, we keep a bucket for each minute worth of check-ins, that bucket is a counter of how many check-ins were consumed for that minute. We will keep these buckets for 7 days worth of data, that's 10080 buckets.

Each time the clock ticks across a minute, we will look at the last 7 days of that particular minute we ticked over, take some type of average of those 7 counts, and compare that with the count of the minute we just ticked past. If we find that the count here is some percentage different from the previous 7 days of that minute, we will produce our clock tick with a "unknown" marker, meaning we are unsure if we collected enough data for this minute and are likely in an incident. In which case we will create misses and time-outs as "unknown".

Ignoring previous incidents

When a minute was detected as having a abnormally low volume we should reset it's count to some sentinel value like -1 so that when we pick up this minute the next 7 days, we know to ignore the data, since it will not be an accurate representation of typical volume.

Not enough data

Warning

What should we do if we don't have enough data to determine if the past minute is within the expected volume?

Implementation

We should start by implementing this as a simple metric that we track, so we can understand what our typical difference looks like each day. It's possible some days may have many more check-ins, such as Monday's ad midnight. So maybe we will need a different way to evaluate anomalies.

Implementation

Tasks

Beta Give feedback

1. feat(crons): Record historic check-in volume counts #79448
   Scope: Backend +1
2. feat(crons): Record stats for volume history at clock tick #79574
   Scope: Backend +1
3. feat(crons): Add "volume_anomaly_result" result to clock-tick sentry-kafka-schemas#339
   +0
4. feat(crons): Add mark_unknown clock task sentry-kafka-schemas#340
   +0
5. feat(crons): Add volume_anomaly_result to mark_missing task sentry-kafka-schemas#341
   +0
6. ref(crons): Bump sentry-kafka-schemas for updated cron tasks #79783
   Scope: Backend +1
7. feat(crons): Begin reporting volume_anomaly_result #79729
   Scope: Backend +1
8. feat(crons): Add mark_unknown clock tick task #79735
   Scope: Backend +1
9. feat(crons): Support dispatching mark_unknown #79785
   Scope: Backend +1
10. feat(crons): Update UNKNOWN status for its new usage #80348
    Scope: Backend +1
11. feat(crons): Skip marking unknowns as failures #80347
    Scope: Backend +1
12. feat(crons): mark misses as UNKNOWN when tick volume is ABNORMAL #80355
    Scope: Backend +1

[evanpurkhiser]

We've realized there is a problem with the approach described in this issue.

As it turns out, it's unlikely we're able to just determine an incident from just looking at volume of a specific clock tick in comparison to volume from the previous clock tick. We did have an incident while building this feature after we've already started collecting metrics for our historic volume, looking at the % deviation from mean during that incident we have the following graph (The second graph being the number of check-ins we processed.

Note that there's a uptick in % mean deviation, followed by a small downtick in % deviation. This downtick is not visible in the actual check ins processed graph and is likely just due to the nature of how the check-in volume variates from minute to minute. So while we did consumer less check-ins at that minute, the deviation is not as great compared to the previous minutes deviation. This does not mean the incident stopped during that downtick, if we were to just use some percentage cut off to determine if a tick is abnormal, we would have processed this tick and likely lost data and create missed check-ins.

This is something we need to account for when determining if we're in an incident state

Given what we've implemented so far, where we make a decision during a tick and will create unknown check-ins instead of missed, and will mark in-progress as unknown. You might intuitively think we can simply delay the click tick and look at future volume deviations during the delayed clock tick, and use that to determine if we are definitely in an incident state, and make the decision for that tick to create misses as unknown etc. But there is a problem with this.

#63166 describes a problem where during a single partition backlog monitors may fail to correctly mark missed check-ins due to to the fact that the clock is delayed, but we are still consuming check-ins from relay. During a clock delay like this, it's possible an OK check-in will come in before we're able to mark missed check-ins for a monitor. By inducing a clock delay for the purpose of detecting incidents, we'll also be inducing this behavior causing real misses to be lost.

Ideally we would just fix this problem, but unfortunately we can't do that. In fact, we are not able to introduce any kind of clock delay without introducing this problem. Even if we were to "fix" this using the proposed solution in that ticket of back-filling missed check-ins, it does not work. Doing this would mean during an incident when we are trying to avoid creating misses and instead create them as unknown, we would still create misses using this backfill. There's no way for them to be backfilled as unknowns since the fact that the clock is delayed (causing this problem in the first place) is because we do not know if we're in an incident yet. So really we would need to delay creating those mises, but that would mean delay processing the OK check in, which we cannot do.

So if we're unable to delay the clock what-so-ever, what can we do?

The problem we're trying to solve for is two fold here

1. Do NOT notify customers of missed and timed-out check-ins during an ingestion incident at sentry. This is THE most important thing, we do not want to wake up customers at 1am when their job ran fine, but we had an ingestion incident.
2. Correctly communicate in our check-in history UI that a missed or timed-out check-in's were created during an incident, and likely do not reflect what their monitor was actually reporting.
3. Be very accurate for when an incident started and when an incident resolved itself by using volume history.

Notifications are the component that needs to be delayed. Specifically notifications for synthetic check-ins (missed, time-out). Once we are sure we are an incident we can mark misses and time-outs as unknown. The question is then how can we delay issue occurrences until we've ticked the clock forward enough to have enough data to determine if we're in an incident and should not produce these notifications?

1. mark_failed is responsible for calling create_issue_platform_occurance. What we can do is instead immediately dispatching the issue occurrence, is to take the arguments and serialize them into an object that we can put into redis. This object will be keyed as monitor-failure-occurrence:{clock_tick_ts}:{monitor_env_id}.

2. We will put that key into a redis set that is keyed as only monitor-failure-occurrences. This set is responsible for tracking all issue occurrences that are pending being sent We will have already created the incidents for these occurrences, but because the occurrences will not yet have been sent no notifications will go out.

3. Finally, we will need to actually dispatch these occurrences. This is when we will determine if we may be in an incident and should delay dispatching the occurrences so we can look at future check-in volume. The actual dispatching will be driven again by the clock tick. Since the clock tick is the source-of-truth for each tick volume, in order to know we have collected enough volume we need to know where the clock is. As the each tick runs, here's what it will do

   1. Get the monitor-fail-occurances set of monitor-failure-occurance:{clock_tick_ts} keys. The clock_tick_ts in the key will always be earlier than the tick that just did this lookup. At this point, we can look at the volume history for the current tick and decide if it has met a very low threshold that we would consider an "incident".
   2. If we have not met the incident threshold we will simply dispatch all the occurrences to send notifications to customers.
   3. If we have met the threshold, we will now introduce a delay in which set of failure occurrences will be processed. This delay will allow for us to look at future clock tick volume and determine if we are entering an incident. Once we've entered an incident we can likely store this in redis and use this for future processing of occurrences to NOT dispatch them.
   4. At this point we can also likely dispatch some type of task to mark misses and time-outs as unknown during the incident.
   5. Once the volume recovers back to a normal amount after some time, we can reset the delay and return to actually dispatching occurrences.

Notes

• Misses are going to show as missed in the UI until they are marked as unknown.
• Timeouts get marked during an incident, how are we able to go back and mark those time-outs as unknowns
  • We can look at the timeout_at and see if that fell within the window for the check-in

It would be nice if we could NOT show the missed and timeouts in the UI until we've dispatch a notification maybe?