Obtain feedback on benchmark tests to compare the performance between OpenTracing and OpenTelemetry

[harshita19244]

Through the benchmark tests, we wish to estimate the overhead/cost of adding instrumentation libraries (OpenTracing and OpenTelemetry) to our applications and effectively compare the performance of the two libraries.

Link to the present test plan:
https://docs.google.com/document/d/17MfQAPDlkWDrp6UDME245Q0DBQ3y6H7z3BH3UQ_u-X8/edit

Link to the present set of OpenTracing tests:
https://github.com/opentracing-contrib/java-benchmarks

Community feedback is highly appreciated to help determine the most relevant set of benchmark tests. Any additional tests that are of relevance can be added to the test plan

[jkwatson]

One thing to consider: OTel doesn't have library instrumentation right now for all of these things. Will this work with -javaagent based instrumentation, I wonder?

[jkwatson]

I suspect they would not be immediately comparable, no. @trask did you mention you have some experience running jmh with -javaagent?

[trask]

hey @harshita19244!

check out https://github.com/glowroot/glowroot/blob/main/agent/benchmarks (e.g. ResultSetBenchmark.java)

IIRC I would run these JMH benchmarks with and without the -javaagent JVM arg in order to measure overhead of javaagent instrumentation.

I think the results will comparable, because once the Javaagent installs the bytecode instrumentation, it should be doing approximately the same work as the library instrumentation. The main additional overhead for the Javaagent is the initial instrumenting of the bytecode, which should only affect the JMH warmup runs.

It would be really great to compare all three:

• OpenTracing (library instrumentation)
• OpenTelemetry library instrumentation (where it exists, e.g. jdbc library instrumentation should land soon! OpenTelemetry JDBC instrumentation library #3367)
• OpenTelemetry javaagent instrumentation

[harshita19244]

This really helps, thank you!
I had a trivial question though, JDBC is listed here as a supported library, so what is the additional JDBC library instrumentation you are working on?
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks

[trask]

that list is for javaagent "auto-" instrumentation

we also are building out "library" instrumentation that people can use if they prefer: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/standalone-library-instrumentation.md

I believe opentracing-contrib generally corresponds to what we call "library instrumentation"

OpenTracing also has a javaagent (called "special agent") for "auto-" instrumentation, in case you have run across that

[jpkrohling]

The special agent wouldn't be comparable to the OpenTelemetry Java Agent, though. I also believed earlier that the list @harshita19244 shared before was also available as libraries to be used individually. If they can only be toggled on/off as a whole, then we might need to evaluate the next steps, @harshita19244 :-)