Broken Kubernetes instrumentation

[mateuszrzeszutek]

I've noticed that our Kubernetes instrumentation is completely broken (and not only because of #2356). It uses an OkHttp Interceptor to wrap the HTTP call with a Span - the problem is, the Kubernetes client interceptor will always run after the OkHttp interceptor (added by the okhttp3 instrumentation). And since the Kubernetes one doesn't check if a CLIENT span is in context we'd get CLIENT->CLIENT span hierarchy (if the propagation wasn't broken 😭 ).

Anyway, here are a few ideas on how to solve that:

• Instrument Kubernetes ApiClient instead to get a higher-level INTERNAL span;
• Change Kubernetes instrumentation so that it renames & adds attributes to the CLIENT span produced by OkHttp instrumentation;
• Just fix it and not care about CLIENT->CLIENT span hierarchy (and ensure that propagation is correct).

WDYT?

[iNikem]

First, I think this should be a proper issue, not a discussion :)

Second, IMO from the modelling POV the pair Kubernetes client/OkHttp is the same as AWS SDK/http client. Meaning that Kubernetes is the higher level API and it should create a CLIENT span with proper semantics of the operation. OkHttp then should detect this and backoff.

[mateuszrzeszutek]

First, I think this should be a proper issue, not a discussion :)

Definitely, but I wanted to discuss how to solve it first.

Kubernetes is the higher level API and it should create a CLIENT span with proper semantics of the operation.

👍 That's more or less what I thought would be the best choice.