Sentry occasionally fails to symbolicate errors even though the required debug symbols are uploaded

[jboulter11]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

The ingestion pipeline clearly will miss occasional errors and they result in a single issue full of events with unsymbolicated traces. While this does not happen super often considering our scale of errors being ingested, it is nonetheless very annoying.

If you look at the provided link, you can see the last event has debug info with a debug id that is listed in the list of uploaded debug info files in Sentry as uploaded 2mo ago.

Expected Result

All traces with uploaded debug files are symbolicated, with none missing symbolication

Actual Result

Some events do not get symbolicated and it results in reopening a ticket every time it happens on a new version.

Product Area

Ingestion and Filtering

Link

https://dropbox.sentry.io/issues/5425338141/

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-processing for triage ⏲️

[loewenheim]

Hi, thank you for the report. In the linked issue the symbolication problem was caused by a 503 service unavailable error when trying to download a debug file from Sentry on Nov. 18. This is probably related to an outage we had that day.

[jboulter11]

That's what we figured. These should not come through as unsymbolicated, reopening this ticket. They should wait to be ingested until they can be. An outage should be absolute: either ingestion works properly or it does not and those events should not be half-ingested, polluting the data. What would it take to make that happen?

[loewenheim]

We're in the early stages of designing improvements to the pipeline, we will definitely keep this issue in mind for that.