Context forking hook where context can be manipulated

[adinauer]

Hey 👋

tl;dr: We're looking for a hook to intercept Context forking, add something to the Context and rely on OpenTelemetry for lifecycle management and propagation. Currently SpanProcessor looks like the only place to do some of that. Is there a way to register an OperationListener globally that I missed or would it be possible to add e.g. a method called addOperationListener to SdkTracerProviderBuilder so we can register global OperationListeners?

---

Update 2024-03-05:

• We noticed that Instrumenter isn't used by users performing manual instrumentation so OperationListener and ContextCustomizer don't solve our problem.
• We tried some more things, listed below
• We think a custom ContextStorage could work for us but we were unable to have it used by Agent and Application

---

I'm working for Sentry and we're looking for a hook that allows us to intercept Context forking. For example, if a new span is created by an OpenTelemetry Instrumenter, it forks the Context, by creating a copy of a Context with that new span set in the Context.

We'd like to register that hook from inside a Java Agent as well as when initializing OpenTelemetry without an Agent. Some of the hooks, that already exist (mentioned below) can only be set on the Instrumenter/InstrumenterBuilder which means they are somewhat hard coded into e.g. TomcatHelper (or more precisely TomcatInstrumenterFactory). We would need something that we can register similar to a SpanProcessor which is then applied to Instrumenter instances globally. So the hook should also be used for OpenTelemetry auto instrumentation, e.g. in case of an incoming request to Tomcat.

The reason for us wanting to hook into Context forking is, so we can create a new Sentry Scope and allow users to manipulate that Scope through a top level static API that automatically goes to the current Scope. We'd like to rely on Context as a way of holding on to that current Scope as well as propagating it to another thread etc. and managing lifecycle for us.

I've looked around OpenTelemetry GitHub repositories and found the following:

SpanProcessor

• This seems to be the only place, where it's currently possible to intercept Context forking unless you can control Instrumenter/InstrumenterBuilder, which is not the case when using OpenTelemetry auto instrumentation. This is limited to spans but at the moment I think that limitation is not a problem for us.
• While we're notified whenever a new span is created or a span has ended in SpanProcessor, we're only handed the parent Context but no place to apply any modifications to it, as possible with other approaches further down.
• This means we cannot use Context to store our forked Scope alongside the current OpenTelemetry span. We have to use some sort of global Map, where we use OpenTelemetry span as weakly referenced key so we don't hold on to memory that should be freed.
• Using span as key should be possible, since SdkSpan implements ReadWriteSpan which in turn implements ReadableSpan.
• This would be our current go to solution, unless you can suggest a better way.
  • To avoid accessing both a global storage and current Context from the application when using our static API, we still require some hook in addition to the SpanProcessor.

SpanPropagator

• This allows us to modify Context in extract but not in inject (we don't mean to suggest changing this).
• extract is only called in cases where the Instrumenter is using a PropagatingFromUpstreamInstrumenter, which is the case for Server / Consumer use case.
• Some Instrumenter instances either don't call extract or don't call anything on SpanPropagator.
• This is fine, I just want to document why we can't use SpanPropagator for what we want to do.

ContextStorageProvider

• We were unable to register a custom ContextStorage via ServiceLoader to be used by both the agent and the application. Only Context storing from the application was intercepted. We also tried registering under the shaded location of the storage provider interface. We found a workaround by using ContextStorageWrappers.addWrapper but that's not public API, so we'd like to avoid that.
• Providing a custom implementation of ContextStorageProvider would allow us to register a custom implementation of ContextStorage that could modify Context before storing it.
• Hands back an OpenTelemetry Scope, which is AutoClosable for managing lifecycle and restoring to the previous Context on close.
• When forking a Context it doesn't necessarily have to be stored (by calling makeCurrent()) so we may never see it in ContextStorage or it could be saved at a later time, causing our Sentry Scope and OpenTelemetry Context to go out of sync. This means we'd probably break OpenTelemetry instrumentation.
  • We now think that's not a problem since OTEL auto instrumentation seems to always call .makeCurrent() (please let us know if it doesn't). For manual instrumentation where the users intent is not to use the current Context we'd have to provide extra API to allow doing this which we can look into at a future time if need arises.
• Could lead to unexpected behaviour, when storing a scope that's different from what the user is calling makeCurrent() on. So doesn't seem like a good idea.
  • This can also simply happen by bytecode manipulated code setting a new Context. Using Context.current() instead of holding a reference to a Context and using that inside library callbacks is probably a better idea.

ContextCustomizer

• This is only available in Instrumenter which manual instrumentation in an application doesn't go through
• This would allow us to modify Context, i.e. return a modified copy that'll then be used for further processing.
• I was unable to find a way to register a ContextCustomizer globally to be used by all Instrumenter instances, when initializing OpenTelemetry, e.g. from an Agent.
• It is also parameterized as ContextCustomizer<REQUEST>, which makes OperationListener seem like the better way.

OperationListener

• This is only available in Instrumenter which manual instrumentation in an application doesn't go through
• Similar to ContextCustomizer but not parameterized.
• Also allows us to return a modified copy of Context to be used for further processing.
• This seems to be precisely what we're looking for.
• I was also unable to find a way to register an OperationListener globally to be used by all Instrumenter instances, when initializing OpenTelemetry, e.g. from an Agent.
• Would it be possible to add e.g. a method called addOperationListener to SdkTracerProviderBuilder so we can register global OperationListeners?

Handing back a wrapped Span from SpanBuilder

• This would allow us to do some work when Span.storeInContext() is called.
• We tried providing a wrapped Span via a wrapped SpanBuilder via a wrapped Tracer via a wrapped TracerBuilder via a wrapped TracerProvider via a wrapped SdkTracerProviderBuilder
  • We locally removed some final keywords from SdkTracerProviderBuilder and SdkTracerProvider
  • SdkTracerProviderBuilder hands back a SdkTracerProvider (not the interface TracerProvider from its build method so we extended SdkTracerProvider but we can't come up with all the params required for the constructor.
  • We also had to move our wrappers for SdkTracerProviderBuilder and SdkTracerProvider into the io.opentelemetry.sdk.trace package due to constructor visibility.

---

Sorry for the lengthy text. Any advice would be much appreciated.

Regards Alex

[laurit]

Do I understand correctly that you wish to do something whenever Scope scope = context.makeCurrent() is called? So when makeCurrent is called you wish to replace the original context with another context that contains extra keys. I assume that you also wish to do something when scope.close() is called. You mentioned using a custom ContextStorage didn't work out for you. Could you elaborate on that.

[adinauer]

Hello @laurit thanks for the answer.

you wish to replace the original context with another context that contains extra keys.

Ideally we'd be interested not only in context.makeCurrent() but also in things like Context.with(...) which I call context forking (as opposed to context storing).

I haven't found an easy way to intercept context forking in opentelemetry-java but after doing some more thinking we now believe it's sufficient to only intercept context storing (i.e. context.makeCurrent()).

We believe that creating a new Context without making it the current one is more of a power user thing where they choose to manage context themselves. If a user does that their modified Context also shouldn't be used e.g. for auto instrumentation in opentelemetry-java. For this advanced use case we'd have to provide extra Sentry API that allows using a specific OTEL Context where we can then most likely also take care of adding/forking our Sentry Scopes.

Where we aren't quite certain is, if storing a modified Context in ContextStorage.attach() is OK to do. For example:

// user has a context they store
Context c1 = Context.current().with("user_context")
c1.makeCurrent()

// our custom ContextStorage now creates a modified copy and stores that
public Scope attach(Context c1) {
	Context c1modified = c1.with("sentry_scopes");
	return delegate.attach(c1modified);
}

// ! c1 is not what actually ended up in the store

Could it become an issue, if we end up storing something different than what the user called .makeCurrent() on?

You mentioned using a custom ContextStorage didn't work out for you.

We have now found ContextStorage.addWrapper() which is public API and calls into ContextStorageWrappers.addWrapper. We use it to add our ContextStorage wrapper that takes care of adding/forking our Sentry Scopes. So I think we've got this part covered now.

I assume that you also wish to do something when scope.close() is called.

I'll have to think about that one a bit more but I think simply relying on OTEL Context propagation and Scope.close cleaning up, we don't necessarily have to intercept it. Should we need it, I guess we could simply return our own wrapper for Scope.

---

We also found that before SpanProcessor.onEnd() is called, the Context is reset to the Context that was active when the span was started. I presume that's by design and you explicitly want the parent Context to be the one returned by Context.current() inside SpanProcessor.onEnd. We weren't able to find a clean way to do our Scopes forking in a way where we can easily retrieve our Scopes from Context in SpanProcessor so we're now planning to use a global storage to work around this. The idea is the following:

Whenever an OTEL span is started, we want to fork our Scopes. While the span is running, we want to be able to retrieve our Scopes from Context. [x] When the span is ended, we want to retrieve our Scopes to apply it to events we generate from the span.

[x] the same process would happen for any child span

Is there something in opentelemetry-java that would allow us to do this? e.g. retrieving Context that was active while the span was executing?

[laurit]

Ideally we'd be interested not only in context.makeCurrent() but also in things like Context.with(...) which I call context forking (as opposed to context storing).

you can probably do this when you provide your own ContextStorage and override https://github.com/open-telemetry/opentelemetry-java/blob/62a4810145eec10abda6b023a70f7e9252ca217c/context/src/main/java/io/opentelemetry/context/ContextStorage.java#L112 then you can return a custom Context that can track calls to with.

We believe that creating a new Context without making it the current one is more of a power user thing where they choose to manage context themselves. If a user does that their modified Context also shouldn't be used e.g. for auto instrumentation

there are use case where making the context current isn't required. For example we have InstrumenterUtil.startAndEnd() that creates a span and immediately ends it. Making the context current only makes sense when you are calling something that might use the current context (e.g. you call a method that may create a child span) while the scope is active.

for auto instrumentation in opentelemetry-java

opentelemetry-java provides api and the sdk, instrumentation is in opentelemetry-java-instrumentation

Where we aren't quite certain is, if storing a modified Context in ContextStorage.attach() is OK to do

as far as I understand it is fine

We also found that before SpanProcessor.onEnd() is called, the Context is reset to the Context that was active when the span was started. I presume that's by design and you explicitly want the parent Context to be the one returned by Context.current() inside SpanProcessor.onEnd

SpanProcessor belongs to https://github.com/open-telemetry/opentelemetry-java questions about it are better asked there. I believe that observing Context.current() returning parent context in SpanProcessor.onEnd is more of a coincidence than anything else. A span can be created by providing a different parent context than what Context.current() returns so using Context.current() inside a SpanProcessor is not guaranteed to give you the parent context. Why you observe the parent context in the onEnd is because then usual pattern in the instrumentation is to close the scope before ending the span. If we did it the other way around you could probably observe a different context in span processor on end callback.

Whenever an OTEL span is started, we want to fork our Scopes. While the span is running, we want to be able to retrieve our Scopes from Context. [x] When the span is ended, we want to retrieve our Scopes to apply it to events we generate from the span.

As the span can be started and ended without adding it to the context tracking the context won't give you access to all the spans, idk whether this would be an issue.

Is there something in opentelemetry-java that would allow us to do this? e.g. retrieving Context that was active while the span was executing?

opentelemetry-java questions are best asked in https://github.com/open-telemetry/opentelemetry-java

[adinauer]

Thanks for the detailed answer!

return a custom Context that can track calls to with.

Oh nice, thanks! I'll take a look and try this.

there are use case where making the context current isn't required. ... For example we have InstrumenterUtil.startAndEnd()

This seems like there's no way for any other code to happen between start and end anyways, so I guess that wouldn't be a problem for us.

SpanProcessor belongs to open-telemetry/opentelemetry-java questions about it are better asked there. I believe that observing Context.current() returning parent context in SpanProcessor.onEnd is more of a coincidence than anything else. A span can be created by providing a different parent context than what Context.current() returns so using Context.current() inside a SpanProcessor is not guaranteed to give you the parent context. Why you observe the parent context in the onEnd is because then usual pattern in the instrumentation is to close the scope before ending the span. If we did it the other way around you could probably observe a different context in span processor on end callback.

Ah right, we're not relying on Context.current() in SpanProcessor.onEnd(), it was just something we noticed and were curious about.

As the span can be started and ended without adding it to the context tracking the context won't give you access to all the spans, idk whether this would be an issue.

Once we want to support this use case, we should just be able to put the span in a Context internally to work around this since users would have to go through our API anyways in this case.