Exporting Span throws class cast exception when artemis queue messages are consumed

[Raushan-Sapiens]

We are using java agent 1.30.0 for service which consumes queue message and have other APIs and exports span to collector. Instrumentations for APIs works well and spans are exported well to the collector.

But when queue messages are consumed, spans are failed to export. I see below error in logs

[otel.javaagent 2024-09-11 06:51:31:602 +0000] [BatchSpanProcessor_WorkerThread-1] WARN io.opentelemetry.sdk.trace.export.BatchSpanProcessor - Exporter threw an Exception
java.lang.ClassCastException: class java.lang.Double cannot be cast to class java.lang.Long (java.lang.Double and java.lang.Long are in module java.base of loader 'bootstrap')
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler$ArrayValueMarshaler.createInt64(KeyValueMarshaler.java:223)
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler.create(KeyValueMarshaler.java:84)
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler.access$000(KeyValueMarshaler.java:30)
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler$1.accept(KeyValueMarshaler.java:48)
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler$1.accept(KeyValueMarshaler.java:43)
at java.base/java.util.HashMap.forEach(HashMap.java:1421)
at io.opentelemetry.sdk.internal.AttributesMap.forEach(AttributesMap.java:88)
at io.opentelemetry.exporter.internal.otlp.KeyValueMarshaler.createRepeated(KeyValueMarshaler.java:42)
at io.opentelemetry.exporter.internal.otlp.traces.SpanMarshaler.create(SpanMarshaler.java:44)
at io.opentelemetry.exporter.internal.marshal.MarshalerUtil.groupByResourceAndScope(MarshalerUtil.java:67)
at io.opentelemetry.exporter.internal.otlp.traces.ResourceSpansMarshaler.groupByResourceAndScope(ResourceSpansMarshaler.java:96)
at io.opentelemetry.exporter.internal.otlp.traces.ResourceSpansMarshaler.create(ResourceSpansMarshaler.java:37)
at io.opentelemetry.exporter.internal.otlp.traces.TraceRequestMarshaler.create(TraceRequestMarshaler.java:32)
at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter.export(OtlpGrpcSpanExporter.java:71)
at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.exportCurrentBatch(BatchSpanProcessor.java:327)
at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:245)
at java.base/java.lang.Thread.run(Thread.java:833)

How can we fix/debug this? Any suggestions please

[laurit]

Can you reproduce this at will? I'd firstly try with the latest version of the agent and see if it changes anything. If the problem persists and you can reproduce this then I'd attach debugger and set exception breakpoint for ClassCastException. Looking at the stack trace it looks like there is an attribute of type long array but the values list contains a double. Debugger could help verifying whether this is the case and the name of the attribute and the span could help to identify where it comes from.

[laurit]

In collector you can only see spans that were successfully exported, but you are interested in a span that fails to export.

[Raushan-Sapiens]

hey @laurit.. Unfortunately not able to create sample code to reproduce with docker compose locally. It works well. Issue only occurs in Azure k8s environment with the same configuration.

Tried to intercept agent exporter and log span attributes but doesn't work well. can you please help in this with any kind of docs OR reference repo.

Any other ideas please. thanks!

[laurit]

Can you reproduce this at will? If you can then you could set the trace exporter to console (see https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters) Assuming that there aren't too many spans I'd imaging that it is possible to find the offending attribute (a list that contains a double). If you can't reproduce it at will then you could write an agent extension that wraps the span exporter using https://github.com/open-telemetry/opentelemetry-java/blob/de13ff183a24aef45dc2df8a4422e612765ea32a/sdk-extensions/autoconfigure-spi/src/main/java/io/opentelemetry/sdk/autoconfigure/spi/AutoConfigurationCustomizer.java#L64 After you have wrapped the exporter before you delegate to the original exporter you can examine the span data and check whether there is an attribute where key is a long array, but values list contains a double. If you find the offending span you can log it. If you decide to go down this route then I'd start by first creating an app that reproduce the issue so you could verify whether your custom exporter can catch it.

[Raushan-Sapiens]

@laurit We identified the root cause of the ClassCastException. The issue stemmed from our GraphQL API, where the schema defined a float data type, but the corresponding Java method was expecting a Long. While this mismatch worked with the GraphQL engine, the OpenTelemetry agent was unable to handle the type conflict, leading to the ClassCastException.

Thanks for the support!

[laurit]

Could you provide more information that would let us figure out which instrumentation produced this so we could see whether we can do something about it.