Don't have transactions go through post_process

[JoshFerge]

given that for post_process we use rabbitmq/celery, and its quite high throughput, we should consider that for transactions, this may not be necessary.

• We don't need anything in clickhouse for transactions processing
• the only thing we do for transactions in post_process is record sampling and send a receive signal.

  sentry/src/sentry/ingest/transaction_clusterer/datasource/redis.py

  Line 103 in 5d39254

  def record_transaction_name(project: Project, event_data: Mapping[str, Any], **kwargs: Any) -> None:

We can cut the number of rabbitmq tasks we send by ~40% if we simply don't use post_process for transactions.

We can also cut out the usage of rc-processing for transactions for this.

We'd move this transaction sampling over to save_event_transaction.

Steps:
Part 1 (Safe Transition):

• create new option "sample_transactions_in_save"
• modify save_event_transaction to do this sampling functionality if the option is set (make sure it works both in celery and kafka version)
• modify post_process to not do the sampling functionality of option is set
• in this step, post process still deletes from the rc-cache
• Flip flag on, confirm behavior looks good in save_event, continue on

I assume that we're okay with a minute of data or so being lost from the transaction clusterer. by doing a simple switch-over, we can save some complexity. (i believe this is just used for accurate dynamic sampling calculations)

Part 2 (Cleanup):

• create new option "post_process_transaction_cleanup"
• modify post_process_transaction to gracefully handle case where transaction not in rc-processing. (this is already the case, marking as completed, no changes needed).
• modify our logic in save_event_transaction to not put the transaction in rc-processing if post_process_transaction_cleanup option is true.
• in the post_process forwarder, stop creating post_process tasks for transactions if post_process_transaction_cleanup option is true

[JoshFerge]

Roll out strategy:

Part 1:

• Merge ref(transactions): add functionality to sample transactions in save_event_transaction #81077
• in S4S, turn on flag ("transactions.do_post_process_in_save") to 1.0. confirm looks good
• in DE, turn on flag to 1%, 10%, 25%, 50%, 100%
• in US, turn on flag 1%, 10%, 25%, 50%, 100%

what to watch for during rollout:

• transaction clusterer metrics (should stay the same)
• save_event_transaction on queue status
• s4s sentry perf page for save_event_transaction
• should also ensure traces for save_event_transaction are reporting the new spans
• no new Sentry errors

Part 2:

Stop producing post_process for transactions

• see that post_process enqueues decline
• see logging messages for "not found" decline

[mwarkentin]

send a receive signal.

@JoshFerge during SRE office hours it was raised that sending signals is not an asynchronous event - basically all of the existing (and future) receivers of those signals are still processed synchronously. So we need to understand what is being triggered by that signal and understand how that impacts things. Are we going to be doing those signal handlers inside of save_event_transaction now too?

[JoshFerge]

send a receive signal.

@JoshFerge during SRE office hours it was raised that sending signals is not an asynchronous event - basically all of the existing (and future) receivers of those signals are still processed synchronously. So we need to understand what is being triggered by that signal and understand how that impacts things. Are we going to be doing those signal handlers inside of save_event_transaction now too?

i've changed the implementation to not use signals and call what the signal did previously directly. we also confirmed that these calls are very fast https://sentry.sentry.io/traces/?project=1&query=span.op%3Atasks.post_process_group.transaction_processed_signal&statsPeriod=24h&visualize=%7B%22chartType%22%3A1%2C%22yAxes%22%3A%5B%22avg%28span.duration%29%22%5D%7D