Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Empty dashboard metrics with OTLP and GRPC Metrics created by OTEL Java agent #14938

Open
NicklasWallgren opened this issue Dec 13, 2024 · 0 comments
Open

Empty dashboard metrics with OTLP and GRPC Metrics created by OTEL Java agent #14938

NicklasWallgren opened this issue Dec 13, 2024 · 0 comments
Labels
bug

Comments

@NicklasWallgren
Copy link

NicklasWallgren commented Dec 13, 2024 •
edited
Loading

OpenTelemetry gRPC metrics sent to the APM Server OTLP endpoint are not displayed correctly in Kibana when originating from the Java OpenTelemetry Agent.

Image

APM Server version (apm-server version):
8.10.4

Kibana version
8.13.2

OTEL Java Agent version:
2.10.0

Expected Behavior:
Should show metrics correctly in Kibana

Steps to reproduce:
Instrument an java application using the OTEL java agent - opentelemetry-java-instrumentation

Metrics Context from OTEL collector

collector-1  | Metric #9
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.gc.duration
collector-1  |      -> Description: Duration of JVM garbage collection actions.
collector-1  |      -> Unit: s
collector-1  |      -> DataType: Histogram
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | HistogramDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> jvm.gc.action: Str(end of minor GC)
collector-1  |      -> jvm.gc.name: Str(PS Scavenge)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Count: 2615
collector-1  | Sum: 5.732000
collector-1  | Min: 0.000000
collector-1  | Max: 0.050000
collector-1  | ExplicitBounds #0: 0.010000
collector-1  | ExplicitBounds #1: 0.100000
collector-1  | ExplicitBounds #2: 1.000000
collector-1  | ExplicitBounds #3: 10.000000
collector-1  | Buckets #0, Count: 2602
collector-1  | Buckets #1, Count: 13
collector-1  | Buckets #2, Count: 0
collector-1  | Buckets #3, Count: 0
collector-1  | Buckets #4, Count: 0
collector-1  | HistogramDataPoints #1
collector-1  | Data point attributes:
collector-1  |      -> jvm.gc.action: Str(end of major GC)
collector-1  |      -> jvm.gc.name: Str(PS MarkSweep)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Count: 4
collector-1  | Sum: 0.930000
collector-1  | Min: 0.074000
collector-1  | Max: 0.362000
collector-1  | ExplicitBounds #0: 0.010000
collector-1  | ExplicitBounds #1: 0.100000
collector-1  | ExplicitBounds #2: 1.000000
collector-1  | ExplicitBounds #3: 10.000000
collector-1  | Buckets #0, Count: 0
collector-1  | Buckets #1, Count: 1
collector-1  | Buckets #2, Count: 3
collector-1  | Buckets #3, Count: 0
collector-1  | Buckets #4, Count: 0
collector-1  | Metric #10
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.cpu.time
collector-1  |      -> Description: CPU time used by the process as reported by the JVM.
collector-1  |      -> Unit: s
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: true
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 155.370000
collector-1  | Metric #11
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.thread.count
collector-1  |      -> Description: Number of executing platform threads.
collector-1  |      -> Unit: {thread}
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: false
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(true)
collector-1  |      -> jvm.thread.state: Str(runnable)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 11
collector-1  | NumberDataPoints #1
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(true)
collector-1  |      -> jvm.thread.state: Str(waiting)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 8
collector-1  | NumberDataPoints #2
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(false)
collector-1  |      -> jvm.thread.state: Str(timed_waiting)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 2
collector-1  | NumberDataPoints #3
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(false)
collector-1  |      -> jvm.thread.state: Str(waiting)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 1
collector-1  | NumberDataPoints #4
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(false)
collector-1  |      -> jvm.thread.state: Str(runnable)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 1
collector-1  | NumberDataPoints #5
collector-1  | Data point attributes:
collector-1  |      -> jvm.thread.daemon: Bool(true)
collector-1  |      -> jvm.thread.state: Str(timed_waiting)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 13
collector-1  | Metric #12
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.memory.used_after_last_gc
collector-1  |      -> Description: Measure of memory used, as measured after the most recent garbage collection event on this pool.
collector-1  |      -> Unit: By
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: false
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Eden Space)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 0
collector-1  | NumberDataPoints #1
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Survivor Space)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 98304
collector-1  | NumberDataPoints #2
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Old Gen)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 97485888
collector-1  | Metric #13
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.memory.limit
collector-1  |      -> Description: Measure of max obtainable memory.
collector-1  |      -> Unit: By
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: false
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Eden Space)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 958398464
collector-1  | NumberDataPoints #1
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(CodeHeap 'profiled nmethods')
collector-1  |      -> jvm.memory.type: Str(non_heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 122912768
collector-1  | NumberDataPoints #2
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(CodeHeap 'non-nmethods')
collector-1  |      -> jvm.memory.type: Str(non_heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 5828608
collector-1  | NumberDataPoints #3
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(CodeHeap 'non-profiled nmethods')
collector-1  |      -> jvm.memory.type: Str(non_heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 122916864
collector-1  | NumberDataPoints #4
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(Compressed Class Space)
collector-1  |      -> jvm.memory.type: Str(non_heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 1073741824
collector-1  | NumberDataPoints #5
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Survivor Space)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 524288
collector-1  | NumberDataPoints #6
collector-1  | Data point attributes:
collector-1  |      -> jvm.memory.pool.name: Str(PS Old Gen)
collector-1  |      -> jvm.memory.type: Str(heap)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 1919942656
collector-1  | Metric #14
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.class.unloaded
collector-1  |      -> Description: Number of classes unloaded since JVM start.
collector-1  |      -> Unit: {class}
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: true
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 1
collector-1  | Metric #15
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.class.count
collector-1  |      -> Description: Number of classes currently loaded.
collector-1  |      -> Unit: {class}
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: false
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 32949
collector-1  | Metric #16
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.buffer.memory.limit
collector-1  |      -> Description: Measure of total memory capacity of buffers.
collector-1  |      -> Unit: By
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: false
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> jvm.buffer.pool.name: Str(mapped - 'non-volatile memory')
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 0
collector-1  | NumberDataPoints #1
collector-1  | Data point attributes:
collector-1  |      -> jvm.buffer.pool.name: Str(mapped)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 0
collector-1  | NumberDataPoints #2
collector-1  | Data point attributes:
collector-1  |      -> jvm.buffer.pool.name: Str(direct)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 8436561
collector-1  | Metric #17
collector-1  | Descriptor:
collector-1  |      -> Name: jvm.class.loaded
collector-1  |      -> Description: Number of classes loaded since JVM start.
collector-1  |      -> Unit: {class}
collector-1  |      -> DataType: Sum
collector-1  |      -> IsMonotonic: true
collector-1  |      -> AggregationTemporality: Cumulative
collector-1  | NumberDataPoints #0
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 32950
collector-1  | ScopeMetrics #1
collector-1  | ScopeMetrics SchemaURL: 
collector-1  | InstrumentationScope io.opentelemetry.sdk.logs 
collector-1  | Metric #0
collector-1  | Descriptor:
collector-1  |      -> Name: queueSize
collector-1  |      -> Description: The number of items queued
collector-1  |      -> Unit: 1
collector-1  |      -> DataType: Gauge
collector-1  | NumberDataPoints #0
collector-1  | Data point attributes:
collector-1  |      -> processorType: Str(BatchLogRecordProcessor)
collector-1  | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1  | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1  | Value: 0

Image
@NicklasWallgren NicklasWallgren changed the title Empty dashboard metrics with OTLP and GRPC Metrics Created by OTEL Java agent Empty dashboard metrics with OTLP and GRPC Metrics created by OTEL Java agent Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@NicklasWallgren