Otel Java agent not exporting System.*, process.*, tomcat.*

[rapar8]

Describe the bug

I setup a project to export metrics to otel-collector via java agent but only certain metrics are being sent.
Not exporting:

process.*,  //Including CPU
system.*,    //Including CPU
tomcat.*

here is my below configuration

Dockerfile

WORKDIR /service-layer
RUN apk --no-cache add curl
RUN curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar --output otel-javaagent.jar
CMD ["java", "-javaagent:otel-javaagent.jar", "-jar","service-layer.jar"]

otel-java-agent-version: io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.7.0

Dependency included:

compile "io.opentelemetry:opentelemetry-exporter-otlp:*"
compile "io.opentelemetry.instrumentation:opentelemetry-runtime-metrics:1.26.0-alpha"
compile "io.opentelemetry:opentelemetry-sdk:*"
compile "io.opentelemetry:opentelemetry-sdk-metrics:*"
compile "io.opentelemetry:opentelemetry-exporter-otlp-metrics:1.14.0"
compile "io.opentelemetry:opentelemetry-api:1.41.0"
compile "io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17:2.7.0-alpha"
compile "io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:2.7.0"
compile "io.opentelemetry.semconv:opentelemetry-semconv:1.26.0-alpha"
compile "org.springframework.boot:spring-boot-starter-actuator:3.3.3"

Steps to reproduce

https://github.com/rapar8/opentelemetry-java-agent

Dcoker compose up

Expected behavior

it should export "process.", "system."

Actual behavior

only produce

http_server_active_requests
http_server_request_body_size_bytes_bucket
http_server_request_body_size_bytes_count
http_server_request_body_size_bytes_sum
http_server_request_duration_seconds_bucket
http_server_request_duration_seconds_count
http_server_request_duration_seconds_sum
http_server_response_body_size_bytes_bucket
http_server_response_body_size_bytes_count
http_server_response_body_size_bytes_sum
jvm_class_count
jvm_class_loaded_total
jvm_class_unloaded_total
jvm_cpu_count
jvm_cpu_time_seconds_total
jvm_gc_duration_seconds_bucket
jvm_gc_duration_seconds_count
jvm_gc_duration_seconds_sum
jvm_memory_committed_bytes
jvm_memory_limit_bytes
jvm_memory_used_after_last_gc_bytes
jvm_memory_used_bytes
jvm_thread_count
otlp_exporter_exported_total
otlp_exporter_seen_total
processedLogs_total
processedSpans_total
queueSize_ratio

Javaagent or library instrumentation version

2.7.0

Environment

• OTEL_EXPERIMENTAL_SDK_ENABLED=true
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:5555
  - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
  - OTEL_IMR_EXPORT_INTERVAL=1000
  - OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
  - OTEL_INSTRUMENTATION_HTTP_CLIENT_EMIT_EXPERIMENTAL_TELEMETRY=true
  - OTEL_INSTRUMENTATION_HTTP_SERVER_EMIT_EXPERIMENTAL_TELEMETRY=true
  - OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true
  - OTEL_INSTRUMENTATION_TOMCAT_ENABLED=true
  - OTEL_JAVAAGENT_DEBUG=true
  - OTEL_JAVAAGENT_ENABLED=true
  - OTEL_JAVAAGENT_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
  - OTEL_JAVAAGENT_LOGGING=application
  - OTEL_LOGS_EXPORTER=none
  - OTEL_LOG_LEVEL=trace
  - OTEL_METRICS_EXPORTER=logging
  - OTEL_METRIC_EXPORT_INTERVAL=1000
  - OTEL_RESOURCE_ATTRIBUTES="job=job,application=omsvclyr-d,service.name=service-layer,service.version=1.0,deployment.environment=development"
  - OTEL_RESOURCE_PROVIDERS_GCP_ENABLED=true
  - OTEL_SDK_ENABLED=true
  - OTEL_SERVICE_NAME=omsvclyr-d
  - OTEL_TRACES_EXPORTER=none
  - OTEL_TRACES_SAMPLER=always_on

Additional context

If I change the otel-java-agent version to 1.28.0. it produce "process.*" "system.*" metrics.

io.opentelemetry.javaagent.tooling.VersionLogger - Running on Java 17.0.12. JVM OpenJDK 64-Bit Server VM - Eclipse Adoptium - 17.0.12+7

[laurit]

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0 For 2.x many metrics that were under process.runtime.jvm were moved to jvm. I'm not sure where the system and tomcat metrics came form, perhaps they are from micrometer/spring boot actuator? If so have a look at migration notes section in https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.1.0 If you decide to enable spring boot actuator instrumentation you may need to disable otel runtime-telemetry instrumentation to avoid duplicate metrics.

- OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
- OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true

imo these flags don't exist

[rapar8]

Thanks for reply...

I am not using actuator/micrometer.

I am looking for metrics apart from JVM under

• System
• OS - process.*
• Container
• Tomcat

I have gone through doc, it says java-agent automatically support many metrics just by setting ENV variable.

Variable name you are asking comes from java-agent and doc, thought if we need to enable if not by default

Note: if I switch to old otel-java-agent.jar(1.28.0) I see all the other metrics.

[SylvainJuge]

For Tomcat, when capturing metrics with JMX Insight, you need to set OTEL_JMX_TARGET_SYSTEM=tomcat to enable them, see docs here for more details and other supported systems.

Also, the non stable JVM runtime metrics are only enabled when OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true, see docs for more details.

[rapar8]

TOMCAT: Update
Below are the ENV fetch from deployed application

OTEL_EXPERIMENTAL_SDK_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:5555/
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_IMR_EXPORT_INTERVAL=1000
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=true
OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED=true
OTEL_INSTRUMENTATION_HTTP_CLIENT_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_HTTP_SERVER_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_JVM_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_PROCESS_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true
OTEL_INSTRUMENTATION_SYSTEM_METRICS_ENABLED=true
OTEL_INSTRUMENTATION_TOMCAT_ENABLED=true
OTEL_JAVAAGENT_DEBUG=true
OTEL_JAVAAGENT_ENABLED=true
OTEL_JAVAAGENT_INSTRUMENTATION_RUNTIME_METRICS_ENABLED=true
OTEL_JAVAAGENT_LOGGING=application
OTEL_JMX_TARGET_SYSTEM=tomcat                      <============
OTEL_LOG_LEVEL=trace
OTEL_LOGS_EXPORTER=none
OTEL_METRIC_EXPORT_INTERVAL=1000
OTEL_METRICS_EXPORTER=otlp
OTEL_PROPAGATORS=tracecontext
OTEL_RESOURCE_ATTRIBUTES="job=job,application=omsvclyr-d,service.name=service-layer,service.version=1.0,deployment.environment=development"
OTEL_RESOURCE_PROVIDERS_GCP_ENABLED=true
OTEL_SDK_ENABLED=true
OTEL_SERVICE_NAME=omsvclyr-d
OTEL_TRACES_EXPORTER=none
OTEL_TRACES_SAMPLER=always_on

May be you can correct me,

Still I could not find the metrics, here is my docker source code -
Execute:
Docker compose up

curl --location 'http://localhost:8080/test/test'
curl --location 'http://localhost:8080/test/env'
check the prometheus metrics in UI - http://localhost:9090/