How to add arbitrary attributes to the exported metrics

[WesselAtWork]

Currently, after collecting the otlp data, I have http_client_request_duration_seconds_sum, http_client_request_duration_seconds_count and http_client_request_duration_seconds_bucket metrics.

They seem to come with the with server_address, job, http_request_method, http_response_status_code, network_protocol_version, server_port labels.

I want to add the url.path to the metrics being exported.

I can see the attribute in the traces.

But more generally, how do we export arbitrary attributes with the metrics?

I might be wrong, this might need to be setup in my collector

[laurit]

But more generally, how do we export arbitrary attributes with the metrics?

you don't

I want to add the url.path to the metrics being exported.

So you wish to add url.path to http.client.request.duration metric. This can be achieved configuring a metric view https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/incubator either programmatically in an agent extension or using a configuration file. Agent limits the set of attributes added to metrics using advice in

opentelemetry-java-instrumentation/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpMetricsAdvice.java

Line 26 in 4dae82c

static void applyClientDurationAdvice(DoubleHistogramBuilder builder) {

metric view can override that advice and change the set of attributes added to the metric.

[WesselAtWork]

Couldn't quite figure out what I was doing wrong

Tried to get a minimal test going with:

- selector:
    instrument_name: 'http_client_request_duration_seconds_sum'
  view:
    attribute_keys:
      - url.path

Tried it on my java application with OTEL_EXPERIMENTAL_METRICS_VIEW_CONFIG=/path/to/file.yaml

I think it's supposed to override all the attributes with the list (in this case url.path, I was planing on adding the rest later), but it didn't seem to do anything for me.

Should I build the incubator artifact and add it to the class path, or is it already included in the javaagent?

If using autoconfigure with this artifact on your classpath

[laurit]

The name of the instrument is http.client.request.duration. I don't think you should add apostrophes or double quotes around the name.

Should I build the incubator artifact and add it to the class path, or is it already included in the javaagent?

It is included in the agent.

[WesselAtWork]

Awesome got it working (kinda)!

url.full works!

- selector:
    instrument_name: http.client.request.duration
  view:
    attribute_keys:
      - url.full
      - http.request.method
      - http.response.status_code
      - error.type
      - network.protocol.name
      - network.protocol.version
      - server.address
      - server.port

Gives me the url.full + all the original attributes advised:

opentelemetry-java-instrumentation/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpMetricsAdvice.java

Lines 33 to 39 in d1fff1d

	HttpAttributes.HTTP_REQUEST_METHOD,
	HttpAttributes.HTTP_RESPONSE_STATUS_CODE,
	ErrorAttributes.ERROR_TYPE,
	NetworkAttributes.NETWORK_PROTOCOL_NAME,
	NetworkAttributes.NETWORK_PROTOCOL_VERSION,
	ServerAttributes.SERVER_ADDRESS,
	ServerAttributes.SERVER_PORT));

url.* don't work

Sadly All the other url.* attributes don't work.

- selector:
    instrument_name: http.client.request.duration
  view:
    attribute_keys:
      - url.full  #this is the only one that works
      - url.path
      - url.domain
      - url.extension
      - url.fragment
      - url.original
      - url.port
      - url.query
      - url.registered_domain
      - url.scheme
      - url.subdomain
      - url.top_level_domain

Workaround

If your request contain query parameters or resource ids, it's going to cause a cardinality explosion.

Of course if it doesn't, it isn't a concern

The workaround is to setup the peer.service with OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING

e.g. OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING="127.0.0.1=test-service"

then doing

- selector:
    instrument_name: http.client.request.duration
  view:
    attribute_keys:
      # your other keys here
      - peer.service

PS

I should probably mention what I needed this for exactly.
I wanted to get stats for my external integration services. These services are not in my control and expose no metrics/tracing/monitoring.
I couldn't use the server.address because they are all funnelled through the same gateway.
Ideally id like to use url.path to differentiate the services, I can use the url.full with regexes to figure it out, but I am afraid of the cardinality explosion.
For me the workaround gives me what I need.

[pepeshore]

Attributes like these are divergent and not recommended to be added to metrics. Moreover, I recall there's a limitation in the SDK that caps the dimensionality combination for a single metric at 3,000. Exceeding this will result in data being discarded, leading to inaccurate metrics.

[trask]

if you know that your url.path attribute values are low-cardinality, you could opt-in to adding them to metrics using https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/incubator#view-file-configuration

[laurit]

@trask http client instrumentation does not fill url.path. To satisfy this request we could provide an extension point that could let user modify attributes in

opentelemetry-java-instrumentation/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpClientMetrics.java

Line 83 in 64bbbc8

duration.record((endNanos - state.startTimeNanos()) / NANOS_PER_S, attributes, context);

Then it would be possible to compute url.path from url.full and a view could be used to include url.path.

[laurit]

Alternatively we could provide an extension point for computing http.route for http client spans where users could could use their knowledge of the system to translate request url to route. We could use that route in span name and also in metrics.

[trask]

we recently added an experimental attribute url.template to HTTP client spans: open-telemetry/semantic-conventions#675

I think this could be a good option here, we just need a way for users to stamp url.template, e.g. similar to open-telemetry/semantic-conventions#1095 (comment)

(or possibly if we think it's common enough for the url.path to be low-cardinality, we could have an opt-in behavior where url.template is populated with url.path)

(cc @lmolkova)

[trask]

I think this could be a good option here, we just need a way for users to stamp url.template, e.g. similar to open-telemetry/semantic-conventions#1095 (comment)

this would be nice: open-telemetry/oteps#207