Incomplete 'http_route' attribute in "http.client.request.duration" metric

[FabrizioFubelli]

I'm having a problem with the opentelemetry-java-instrumentation.

The attribute http_route in the http.client.request.duration metric, is incomplete for all the requests having a 404 or a 405 server response:

Case 01: 405 Method Not Allowed

Instrumentation Output:

• spring-webmvc-6.0

  http_server_request_duration_seconds_sum{
    http_request_method="POST",
    http_response_status_code="405",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.spring-webmvc-6.0",
    otel_scope_version="2.6.0-alpha",
    url_scheme="http"
  } 0.298885301
  

• tomcat-10.0

  http_server_request_duration_seconds_sum{
    http_request_method="POST",
    http_response_status_code="405",
    http_route="/example-service/*",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.tomcat-10.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 1.14099499
  

• servlet-5.0

  http_server_request_duration_seconds_sum{
    http_request_method="POST",
    http_response_status_code="405",
    http_route="/example-service/*",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.servlet-5.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 0.912806257
  

• micrometer-1.5

  http_server_requests_seconds_sum{
    error="none",
    exception="none",
    method="POST",
    otel_scope_name="io.opentelemetry.micrometer-1.5",
    outcome="CLIENT_ERROR",
    status="405",
    uri="UNKNOWN"
  } 0.2612307
  

Problems:

Here the Instrumentation has the following behaviour:

• "spring-webmvc-6.0" does not provide the http_route attribute
• "tomcat-10.0" and "servlet-5.0" provide an incomplete http_route attribute, by showing only the context path
• "micrometer-1.5" shows uri="UNKNOWN"

Case 02: 404 Page Not Found

Instrumentation Output:

• spring-webmvc-6.0

  http_server_request_duration_seconds_sum{
    http_request_method="GET",
    http_response_status_code="404",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.spring-webmvc-6.0",
    otel_scope_version="2.6.0-alpha",
    url_scheme="http"
  } 0.366664799
  

• tomcat-10.0

  http_server_request_duration_seconds_sum{
    http_request_method="GET",
    http_response_status_code="404",
    http_route="/example-service/**",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.tomcat-10.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 0.264330906
  

• servlet-5.0

  http_server_request_duration_seconds_sum{
    http_request_method="GET",
    http_response_status_code="404",
    http_route="/example-service/**",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.servlet-5.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 0.479881698
  

• micrometer-1.5

  http_server_requests_seconds_sum{
    error="none",
    exception="none",
    method="GET",
    otel_scope_name="io.opentelemetry.micrometer-1.5",
    outcome="CLIENT_ERROR",
    status="404",
    uri="/**"
  } 0.371276541
  

Problems:

Here the instrumentation look like the Case 01, excepts for "micrometer-1.5":

• "spring-webmvc-6.0" does not provide the http_route attribute
• "tomcat-10.0" and "servlet-5.0" provide an incomplete http_route attribute, by showing only the context path
• "micrometer-1.5" shows uri="/**"

Note

As you can see, I tried with multiple instrumentation, but it seems that whenever the client makes a wrong request, the OpenTelemetry Java agent is not able to detect the full http_route.

Instead, in case of exceptions in the application logic, or http status 200, the complete http_route attribute is available:

Case 03: Exception in a custom RequestHeaderAuthenticationFilter class

• servlet-5.0

  http_server_request_duration_seconds_sum{
    error_type="500",
    http_request_method="GET",
    http_response_status_code="500",
    http_route="/example-service/api/get-something",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.servlet-5.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 0.003682102
  

• micrometer-1.5

  http_server_requests_seconds_sum{
    error="PreAuthenticatedCredentialsNotFoundException",
    exception="PreAuthenticatedCredentialsNotFoundException",
    method="GET",
    otel_scope_name="io.opentelemetry.micrometer-1.5",
    outcome="SERVER_ERROR",
    status="500",
    uri="/api/get-something"
  } 0.0021745
  

Case 04: HTTP status 200

• servlet-5.0

  http_server_request_duration_seconds_sum{
    http_request_method="GET",
    http_response_status_code="200",
    http_route="/example-service/api/get-something",
    network_protocol_version="1.1",
    otel_scope_name="io.opentelemetry.servlet-5.0",
    otel_scope_version="2.7.0-alpha",
    url_scheme="http"
  } 2.001230406
  

• micrometer-1.5

  http_server_requests_seconds_sum{
    error="none",
    exception="none",
    method="GET",
    otel_scope_name="io.opentelemetry.micrometer-1.5",
    outcome="SUCCESS",
    status="200",
    uri="/api/get-something"
  } 1.956642562
  

Questions

1. Is it possible to get the entire http_route also in the cases "01: 405 Method Not Allowed" and "02: 404 Page Not Found" ?
2. Is it possible to customize a metric like http.client.request.duration, by adding a new attribute?
   Even if it should be taken from the server response body?

[laurit]

Semantic conventions require for http.route to have low cardinality, see https://opentelemetry.io/docs/specs/semconv/attributes-registry/http/ This means that for 404 it either can't be filled or can be filled with something generic like /context-path/*.

[FabrizioFubelli]

Ok, but why only 404 and 405 have this behaviour?

[laurit]

How do you produce a low cardinality route for 404? The user could have requested anything.

[FabrizioFubelli]

It's enough to don't show the parameters in my case, anyway in DataDog those routes where always shown.

[laurit]

Metric attributes should have a low cardinality. Using a high cardinality component is problematic because metrics are identified based on unique combination of attributes. So if you have a metric with a single attribute where the attribute has 10 possible values, vendors will count it as 10 metrics. If a metric has 2 attributes one with 10 values and other with 5 values then it will be counted as 50 metrics. If your vendor charges $0.01 per metric and you have a metric with http request path as attribute and someone requests 100M unique paths you'd end up with a $1M bill.

[jaydeluca]

As a real world example, we had a custom metric tracking URL as an attribute, and we had a cardinality explosion due to pentesting where they were calling all kinds of non existent resources which blew up our bill