[Epic] Extrapolation for extracted metrics

[jjbayer]

SDKS apply a client-side sampling rate to transactions / spans, so any metrics we extract from them are inherently sampled, even with server-side sampling disabled. To represent the sampling rate, take the client-side sampling rate from the envelope header and add it as a weight to any metric bucket extracted from the transaction / span payload (weight := 1.0 / dsc.sample_rate). When merging two buckets, add their weights together.

Tech Spec

https://www.notion.so/sentry/Metrics-Extrapolation-520ab136cb7d48ebb77be9a538bdef91

Open question: data duplication

Ideally, storage supports these weights as well. If / as long as storage does not support weights, we could convert weights into repeated data in Relay.

• Counters: Counters are floats, so simply multiply the weight by the counter value when merging.
• Gauges: Similar to Counters.
• Distributions: need to repeat entries to simulate the weight. Doing this in (or right before) the Kafka producer would minimize rounding errors.
• Sets: Questionable if we can extrapolate in a mathematically sound way. Simplest approach would be to ignore weights. Literature:
  • https://www.vldb.org/conf/1995/P311.PDF
  • https://www.cs.cmu.edu/afs/cs/user/dwoodruf/www/knw10b.pdf
  • https://arxiv.org/pdf/2402.08604

Preparation

Beta Give feedback

1. Prepare the spec #3738
   +0
   

Custom Span Attributes

Beta Give feedback

1. feat(server): Extrapolate sampled metrics numbers #3753
   +0
   
2. feat(settings-metrics): Add project/organization option extrapolatedMetrics sentry#73289
   Scope: Backend +1
3. Project-level override for extrapolation sentry#73185
   +0
   
4. Expose extrapolation to Relay via project config sentry#73623
   +0
   
5. https://github.com/getsentry/team-ingest/issues/366

Internal Test: Spans and Transactions

Beta Give feedback

1. Add a temporary setting for transaction / spans metrics
2. https://github.com/getsentry/team-ingest/issues/369
3. Org-level toggle to opt into extrapolate sentry#73186
   +0
   
4. https://github.com/getsentry/team-ingest/issues/365

Public: Spans and Transactions

Beta Give feedback

1. Rollout to SaaS + DE
2. Update defaults for new projects (epoch)

[iambriccardo]

I am not an expert but given two buckets b1 and b2 with weights w1 and w2, if you sum w1 + w2 during merge you will not get the equivalent extrapolation as if you were to compute it correctly as as 1.0 / ((b1 * w1) + (b2 * w2) / (b1 + b2)).

I may be missing something though.