Add OTLP log endpoint to convert log events to span annotations

[codefromthecrypt]

Feature

This would add a configurable log listener, which would convert log events to span annotations. It could make some heuristic decisions to say tell the difference between a normal log record and a log event. Also, it could make a decision on if it drops the bodies, or attaches them to the span as tag values.

This should be an option, not always on, as certainly some will be fine to use the existing approach of storing logs in ElasticSearch and using UI properties to link to a query.

Rationale

This helps as currently span events are migrating to log events, which dramatically impacts navigation in zipkin. For example, you would need to setup log correlation and jump screens to get to them, vs now when annotations are in the same UI.

Technically, the out-of-order concern isn't a problem in zipkin as out-of-order data has been a norm since day one. Specifically the problem of span annotations sent before or after the span isn't an issue.

Example Scenario

Notably, this is just now happening with OpenAI which moved from span events to log events.

open-telemetry/opentelemetry-python-contrib#3006

Prior Art

None so far as there's no work on something like this to my knowledge.

[making]

reference:

• Add LogRecordProcessor to record event log records as span events open-telemetry/opentelemetry-java-contrib#1551
• https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-openai-v2/examples/zero-code

[making]

@codefromthecrypt It this UI something you are expecting?

the trace data is 4d5f222a01f81091be15d76e2af0c528.json.

[codefromthecrypt]

@making yeah. there are some choices we could make with regards to truncation, or dropping the large event payload (e.g. replace it with event.name), but this is the start indeed and for beginners it is much better than dropping the log events entirely.

Another note is that while python is using the "Event api" other languages are likely to skip that and just write logs directly. For example, this one does that https://github.com/elastic/elastic-otel-node/tree/main/packages/instrumentation-openai