How to use custom implementation of ConfigurableSpanExporterProvider for java auto agent

[patsonluk]

Hi all,

We are trying to provide our own implementation of ConfigurableSpanExporterProvider, with the corresponding META-INF file to list our implementation as a service provider.

However, we cannot seem to get our provider to load properly via -Dotel.traces.exporter

The issue seems that https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/SpanExporterConfiguration.java#L34 cannot load our provider implementation. (tested with the java auto instrumentation agent 1.0.0)

We tried:

1. Adding our exporter implementation jar into the normal application class path as a maven dependency - but the context loader (io.opentelemetry.javaagent.bootstrap.AgentClassLoader) cannot locate it . Probably due to the AgentClassLoader has a different lookup from regular app lookup
2. Adding our jar to the bootstrap classpath - the class itself is actually visible, but it failed to resolve the interface io.opentelemetry.sdk.autoconfigure.spi.ConfigurableSpanExporterProvider, which makes sense as the bootstrap loader probably does not know io.opentelemetry.sdk.autoconfigure.spi.ConfigurableSpanExporterProvider from the javaagent jar (not 100% sure, didn't dig deep into the class loading logic)

We could still use -Dotel.javaagent.experimental.exporter.jar to make things work, but we are wondering what is the correct ConfigurableSpanExporterProvider approach here?

Many thanks for your kind attention!

[jkwatson]

HI @patsonluk . I think this is a question for the opentelemetry-java-instrumentation project, since it's specifically about loading SPI implementations into a javaagent.

I'm assuming that your provider works fine without the agent in place, if you're just using standard autoconfigure to build an SDK?

[trask]

hey @patsonluk! you can use otel.javaagent.experimental.initializer.jar to add your jar to the agent class loader, and then it should pick up your autoconfigure SPI(s), keep in mind this will change a bit (but use case will still be supported) in #2442

[patsonluk]

Thank you @trask !

We tried -Dotel.javaagent.experimental.initializer.jar=<our jar>

It seems to locate our class fine, but it triggers exception like below:

ERROR io.opentelemetry.javaagent.OpenTelemetryAgent
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:64)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.startAgent(AgentInitializer.java:44)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:30)
	... 12 more
Caused by: java.lang.NoClassDefFoundError: io/opentelemetry/api/common/AttributeKey
	at com.appoptics.api.ext.AppOpticsSampler.<init>(AppOpticsSampler.java:24)
	at com.appoptics.api.ext.AppOpticsSamplerProvider.createSampler(AppOpticsSamplerProvider.java:12)
	at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.lambda$configureSampler$0(TracerProviderConfiguration.java:128)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureSampler(TracerProviderConfiguration.java:125)
	at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:37)
	at io.opentelemetry.sdk.autoconfigure.OpenTelemetrySdkAutoConfiguration.initialize(OpenTelemetrySdkAutoConfiguration.java:45)
	at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installAgentTracer(OpenTelemetryInstaller.java:36)
	at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.beforeByteBuddyAgent(OpenTelemetryInstaller.java:27)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installComponentsBeforeByteBuddy(AgentInstaller.java:167)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:101)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:85)
	... 18 more
Caused by: java.lang.ClassNotFoundException: io.opentelemetry.api.common.AttributeKey
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 38 more

We are trying to supply our custom Sampler and Exporter and both has some static constant fields that reference io.opentelemetry.api.common.AttributeKey which cannot be loaded here. Looks like some classloading issues :( . If we remove those static fields then our Sampler and Exporter instantiates fine, but of course it later on still trigger exceptions when methods are invoked (which we moved the AttributeKey to)

Any insights please?

[trask]

@patsonluk oh yes, you will need to shade OpenTelemetry API to match the shading done in the javaagent, in particular something like:

relocate "io.opentelemetry.api", "io.opentelemetry.javaagent.shaded.io.opentelemetry.api"
relocate "io.opentelemetry.semconv", "io.opentelemetry.javaagent.shaded.io.opentelemetry.semconv"
relocate "io.opentelemetry.spi", "io.opentelemetry.javaagent.shaded.io.opentelemetry.spi"
relocate "io.opentelemetry.context", "io.opentelemetry.javaagent.shaded.io.opentelemetry.context"

[anuraaga]

By the way, we have this issue to track the entire topic of extending the agent

#2442

I suspect we'll go with runtime shading to make adding extensions easier so you don't need to reproduce our shading configuration.

[patsonluk]

@trask It's working now! Many thanks!

And I agree with @anuraaga that runtime shading will make it easier for extensions :)