Structured logging in Slf4J not honoured in log entry Body

[marcoboi]

Hi all,

I apologize in advance if this is a known issue, but I could not find any entry clearly describing this issue.

Describe the bug
When a Java application using Slf4j is instrumented, the logs are sent to the OpenTelemetry Collector.
Slf4j provides support for structured logging:

logger.atInfo()
        .addKeyValue("a_key", "a_value)
        .addKeyValue("another_key", "another_value)
        .log();

This should result in a log entry with the following Body:

{
    "a_key": "a_value",
    "another_key": "another_value"
}

While the log entry is sent to the OpenTelemetry Collector and all the attributes are correctly attached, the Body in the log entry is empty.

Steps to reproduce
Java App dependencies

dependencies {
    implementation("org.slf4j:slf4j-api:2.0.13")
    runtimeOnly("ch.qos.logback:logback-classic:1.5.6")
}

Java App code

Logger logger = LoggerFactory.getLogger(App.class);
logger.atInfo()
        .addKeyValue("a_key", "a_value)
        .addKeyValue("another_key", "another_value)
        .log();

Auto Instrumentation
Image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.33.5

Inspecting Logs
Log entries sent by the application are intercepted at the level of the Collector and printed using a debug exporter.

What did you expect to see?
Log entries streamed from the app should contain a body consisting of a json object containing the logged fields:

{
    "a_key": "a_value",
    "another_key": "another_value"
}

What did you see instead?
The Body field for the log entry is empty.
The fields to not appear anywhere in the logs.

What version and what artifacts are you using?
Auto instrumentation Image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.33.5

[jkwatson]

since this is instrumentation related, moving to that repo

[laurit]

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/logback/logback-appender-1.0/javaagent/README.md There is an option to capture key value pairs as attributes.

Log entries streamed from the app should contain a body consisting of a json object containing the logged fields:

I just wanted to point out that even if these attributes were added to the body then it is unlikely to be as json but rather some sort of protobuf encoding.

[marcoboi]

Hi @laurit,
thanks for chiming in there!

I see the configuration you linked uses system properties.
Are environment variables also supported for passing in that configuration?
I'm using auto-instrumentation with the Otel operator, so I don't think I can easily pass in system props.

Regarding your remark about the format of the body, I seem to understand emitting the body (to be clear, I'm referring to the otel data model) in structured format is not supported.
Is my understanding correct?
And if so, what are the reasons behind that choice?

[laurit]

I see the configuration you linked uses system properties. Are environment variables also supported for passing in that configuration? I'm using auto-instrumentation with the Otel operator, so I don't think I can easily pass in system props.

Environment variables are supported.

Regarding your remark about the format of the body, I seem to understand emitting the body (to be clear, I'm referring to the otel data model) in structured format is not supported. Is my understanding correct? And if so, what are the reasons behind that choice?

Java api currently only support string body, see open-telemetry/opentelemetry-java#6591 Even if that gets resolved we'd need to decide how exactly are we going to structure the body. Enabling structured body by default is probably going to be a breaking change and may require major version change. An additional concern is whether backends are going to be able to make sense of the structured body.

[marcoboi]

Understood, thank you for your help @laurit.
My hope was indeed to be able to pass that body as a structured message.
I'll watch out for that ticket you pointed me to.