Connection Pool Timeout Issues During Batch Processing with OpenTelemetry Agent

[leon4652]

java agent version : 2.4.0, use custom collector using OTLP/gRpc

Hello,
I am writing to report an issue encountered during the instrumentation process on a production server.

1. Context

We are operating a live server, and as the date progresses, there is a batch operation that triggers frequent DB connections within a short period. When using the OpenTelemetry agent during this process, we face a CP timeout issue during the batch insert operations, leading to failures in the server’s core logic.

2. Cause

We suspect that the cause of this issue is due to the delay in returning the connection to the Connection Pool, which is caused by the instrumentation process of the agent’s interceptor. This delay seems to occur due to the trace generation and the insertion of data into the JVM’s internal agent buffer (queue).

To elaborate, I hypothesize the following sequence of events:

1. A rapid increase in measured data within a short time.
2. A large volume of inserts is triggered after data generation.
3. The JVM agent buffer gets filled.
4. The buffer gets filled before the periodic batch transmission to the collector, causing a delay in data insertion while waiting for the batch to be sent (assumed).
5. This delay in the data insertion process results in the delayed return of the connection during the transaction.

3. Actions Taken and Request

Referring to the documentation at https://opentelemetry.io/docs/languages/java/configuration/, I have attempted to modify the bsp.* options, specifically the Batch Span Processor options, by increasing the batch size and speeding up the transmission. However, the CP timeout exception still persists.

I reviewed the documentation but could not find detailed explanations or parameters regarding the storage of trace data in the JVM buffer after generation, which is why I am raising this issue.

I would like to understand whether, under the default options, the agent waits for space to become available in the buffer when it is full, or if it discards data when the buffer is full. Additionally, are there any parameters or documentation available to configure this behavior? Your assistance would be greatly appreciated.

Thank you.

[leon4652]

Hello, I’m posting a troubleshooting response for other similar cases. There were several misconceptions on my part, and this is the process of resolving them.

1. Basically, in the BatchSpanProcessor (BSP), if the buffer is full, the while loop inside the run method drops the data. That is, when the queue is full, any additional collected data will be dropped.
2. The actual data is moved to the batch queue by the flush operation, and during this process, the actual span data transmission occurs.
3. To reduce the load of data generation, we increased the size of the transmission batch queue for a short period of time, accepting the IO overhead (from 2048 to 8096, with 1000ms per batch), and reduced the measurement scope (hibernate, jdbc remove).
4. Additionally, we increased the maximum gRPC collection size on the collector side.