Tracing Reactive publishers with multiple subscribers (Reactor, RxJava)

[HaloFour]

I've been looking into ways to enhance the async strategies for reactive publishers (Reactor, RxJava) returned from traced method so that they can handle multiple subscribers. Currently the strategies intercept the publisher and wire up completion notifications that only support ending the span that was started when the method was first called. If the publisher ends up being reused and there are multiple subscribers there is no further tracing information captured. As publishers are typically cold and resubscription often results in reevaluation of the reactive chain (including upstream HTTP requests) I think there's value in supporting additional spans created around those subscriptions.

I've looked into refactoring the Reactor strategy so that instead of composing over the doOnSignal methods to wire up completion notification that it uses an operator to intercept each subscription so that it can replace with the subscriber with an implementation that can intercept those same signals to end the span. This approach also allows intercepting subsequent subscriptions so that it would be possible to use the same Instrumenter instance to start a new span which would then be ended when that subscription gets a completion signal. I've created draft PR #3751 to explore this refactoring. I've looked into RxJava 2/3 and it supports a very similar approach, although they have a different set of operator/observer interfaces per publisher making it a bit more verbose.

My opinion is that the approach is worthwhile, but I wanted feedback around it plus whether additional information should also be captured. For example, an attribute that can indicate whether the span represents the method+subscriber or a subsequent subscriber. I was thinking that events might be useful as well to indicate when the method returned, when a subscription was made, etc. All could be controlled via configuration, as the experimental attributes for cancelation are now.

[iNikem]

On a first glance this seems like a sensible thing to do. One question that immediately came to mind is how to model paren-child relationship for subscribers spans. Although this may be a totally separate problem...

[HaloFour]

That's a good question. In this example implementation for Reactor I take the current context at the time of subscription as the parent context when creating a new span so that it will be a part of the trace of the subscriber. That feels right to me but I can see there being some value in at least a link back to the parent span that created the publisher.