Mapping Otel traces levels to SEQ levels

[antezecevic]

Hi @nblumhardt,

We started using SEQ as the Otel Tracing and Metrics (soon) collector, and we noticed that there is pretty big issue with the log levels and how they are used within SEQ.

As you know, the standard Otel levels are OK and ERROR, but these are not mapped to Information and Error levels that SEQ uses. Therefore, when we use the API Keys with Minimum level set to anything other than empty the traces are not accepted by SEQ.

This is a pretty big usability issue because now we have to create duplicate keys for all services manually, to be able to use key1 for logs and key2 for traces and metrics.

We looked into possible solutions, but nothing realistic pops up:

1. Transform levels from Otel to different values: This doesn't seem possible, at least not with out of the box Otel libraries.
2. Transform levels when they are received by SEQ: We can create a SEQ APP that would transform these, but it seems that the events are passed to the app only after they pass the minimum level checks, which in our case would be never.
3. Wait for native support for this in SEQ, so that we can select OK and ERROR in the list of levels in SEQ, or that traces and metrics endpoint do not honor the minimum levels of an API key when it can't be configured properly.

Do you have any recommendations or ideas how this can be achieved without creating duplicate tokens for all services (we have thousands)?

There is a similar discussion for this, but it addresses nothing: #2175

Thanks!

[nblumhardt]

Thanks for dropping us a line @antezecevic. This is a duplicate of #2213.

In the interim, instead of using the minimum level drop-down control, you can use a filter instead - e.g.

@Level in ['error', 'critical', 'fatal', 'warning', 'warn', 'information', 'info', 'ok'] ci

(The filter box is the next one down on the form in the screenshot.)

We'll roll out a maintenance release for 2024.3 in the next few weeks which addresses this.

[antezecevic]

Hi @nblumhardt,

With the filter option the events are discarded when they arrive in SEQ, unlike the minimum level option which is detected and honoured by the applications before sending the logs. This will cause a lot of unnecessary load on the applications and the SEQ server.

Do you know if the maintenance release 2024.3 do this properly? Does it include the filter in the log level switch so the application that is sending logs can react on this, or will it transform this somehow to the seq sink compatible levels?

I am asking because we can't use the filter as recommended due to the increased load on the app.

Thanks!

[nblumhardt]

Hi @antezecevic,

The maintenance release version will consider Ok to be the same as Information for ingest level control, and will work as expected with the remote application, where client-side levelling is supported. Should be a week or so away - we'll have this out as soon as possible.

Best regards,
Nick

[antezecevic]

Great, thanks a lot for the quick answer Nick!