High CPU during Startup with OTel Java Agent

[suman2007]

When I instrument my service with Java Agent the CPU utilization gets almost more than double 530% during startup of service and last for 10 minutes, after that it was reduced and became normal.
Is the behaviour is expected?
How to tune Java Agent to reduce the CPU utilization?

[steverao]

When I instrument my service with Java Agent the CPU utilization gets almost more than double 530% during startup of service and last for 10 minutes, after that it was reduced and became normal. Is the behaviour is expected? How to tune Java Agent to reduce the CPU utilization?

Which Java Agent version you used? Does this phenomenon exist in an application or does it occur when testing many applications?

[suman2007]

I am using Java Agent Version 1.32.0.
I have tested with two different applications, and in both applications, the outcome is the same high CPU during startup and persists for 10-14 minutes. After that, the CPU was getting down slowly to 15%.

[steverao]

The percentage of CPU utilization is hard to describe the overhead. Can you tell us how many cores increase and that is common performance in other application or not?

[suman2007]

My system has 46 cores. There is not any load in the service. CPU% was calculated with ps -p $pid -o %cpu
Initial CPU%
Service: 401
Service+Agent 436

                   **After 15mints**

Service: 14%
Service+Agent 21%

[steverao]

Java Agent is used to modify your application's bytecode and add some aspect logics. Its overhead is influenced by many factors, such as the number of classes, QPS, and etc. So at first, if your app has many classed need to initialize, it will take up more resources than usual. I think Java Agent should be normal and you had better to find the root cause why your app takes so much CPU resource when it starts. Hi @laurit, for the problem, WDYT? Whether we have a benchmark report of Java Agent before?

[suman2007]

@steverao Absolutely correct, I am sharing a report with a spring boot application with minimum code having only 1 class.

Without Agent Initial CPU%: (299+294+285)/3 = 293
With Agent Initial CPU%: (309+298+297)/3 = 301
~ 8%

The query is that if I have 10 services hosted as containers then overall overhead in the Host system will increase by 80% during startup.

[steverao]

Use Java Agent will have some extra overhead for each service, I think you can calculate the approximate overall overhead of 10 services according to way you mentioned above and reserve some system resources when you use it.