Include any zipkin-capable JVM agents/systems

[codefromthecrypt]

JVM agents are used to trace existing code without modification to libraries or application configuration. We often get asked about this, but it is a tough one as usually agents are a part of a larger APM system where tracing is one of many features. That said, we should be able to answer the question, when it is asked..

Are there any zipkin-capable agents (eg agents that can report data to zipkin)?

Here are a few that are sometimes implicated with zipkin, but I don't think their agents actually support reporting to zipkin at the moment. In almost all cases zipkin is a small or smaller set of APM features, so reporting to zipkin could not obviate the normal backends, rather just avail another view.

• StageMonitor - StageMonitor includes a java agent and there's experimental support for Zipkin. cc @felixbarny
• Hawkular - Hawkular supports reporting to zipkin, but not via its agent. cc @pavolloffay
• Instana - Instana uses ByteBuddy to instrument JVM apps. Some early work used Brave I think, but I don't think it uses Brave or can report to zipkin cc @raphw @CodingFabian
• Pinpoint - Pinpoint has an agent which includes distributed tracing functionality, but it is a different format and reports only to the pinpoint server. cc @Xylus

I'm not sure if there are others, or if above is 100% accurate. We should probably list agents on the website who have long-term interest in supporting Zipkin (ex those test zipkin works independent of side-effect of reporting libraries).

[felixbarny]

That's right. I'm currently working on Zipkin support for stagemonitor which uses Byte Buddy and runtime instrumentation. That means that the agent attaches automatically without the need of specifying the -javaagent flag. Having stagemonitor in the classpath is enough. It also offers a lightweight profiler which is neat for development and collects metrics which can be accessed in Grafana dashboards

[CodingFabian]

you are correct. Instana used Brave once upon a time. Since then our internals evolved quite a bit, and we found many of the concepts in Brave not matching up with our ideas.
We also do allocation free tracing, which we found Brave very difficult (and compatible for other brave users) to migrate to.

As of now, we cannot send to zipkin, and this will probably not change unless a customer requests it. It is worth noting that our backend can handle hundreds of thousands of spans per second, something we found zipkin doesn't do well. So its unlikely we will ever do that.

What we however are currently evaluating is accepting Opentracing and Zipkin Traces / Implement OpenTracing APIs to consume additional spans sent to us. We already have a webservice (https://github.com/instana/instana-java-sdk/blob/master/README.md#instana-trace-webservice) for arbitrary spans, so I assume that is doable in the near future.

We checked, and our agent can coexist with manual Brave instrumentation for example.

[pavolloffay]

Hawkular APM has Java agent implementation, but it sends data in Hawkular APM format, however we support collecting data from zipkin instrumentations.

[nkarthik86]

@felixbarny @adriancole I am looking to use StageMonitor to instrument a JVM and then send this data to zipkin.

Could you please tell me how do I achieve this?

[felixbarny]

Hi @nkarthik86,

this is currently not possible in the current version of stagemonitor, however I'm currently implementing support for that.

[codefromthecrypt]

looks like there's progress on stagemonitor stagemonitor/stagemonitor#254

[yanglifan]

I just want to ask a basic question in here. What are pros and cons of Java agent way to implement a tracing system (or APM system), compare with the interceptor way used by Brave and Spring Cloud Sleuth. Will the Java agent way be error-prone compared with the interceptor way since the former is more complex?

[felixbarny]

The advantage is that the configuration is much simpler: You don't need to for example manually wrap HTTP client libraries, register servlet filters, modify your JDBC URLs etc. Also, in contrast to Spring Cloud Sleuth tools like Instana or stagemonitor are not specific to a particular framework like Spring but can potentially work with a wider range of technologies.

Regarding the question if its more error prone or complex: It certainly can be more complex to deal with raw byte code, but things have changed in that area lately. Tools like Byte Buddy abstract most of the low level byte code stuff so that its almost impossible to produce invalid byte code and cause your application to crash. Also, usually, agents like stagemonitor don't inject tons of byte code into your application but only do small things like registering a plain Java wrapper class for JDBC connections. This plain java wrapper then takes care of the actual instrumentation. Instana wrote a good blog describing how they use Byte Buddy to safely instrument applications: https://www.instana.com/blog/how-instana-safely-instruments-applications-for-monitoring/

Disclaimer: I'm the author of stagemonitor

[codefromthecrypt]

zipkin support is published with census (dapper successor) starting with version 0.8. Agent support just recently started census-instrumentation/opencensus-java#780

[wu-sheng]

• Apache SkyWalking : SkyWalking has a java agent. It is used its own format and reports only to SkyWalking collector only. The format is open. We are planning to support zipkin's format later.

[alex-lx]

@wu-sheng How is the progress?

[wu-sheng]

@alex-lx SkyWalking could work with Zipkin trace format, which means SkyWalking could receive and process zipkin instrument data.

[alex-lx]

@wu-sheng will SkyWalking agent support reports to zipkin server?

[wu-sheng]

@alex-lx right now, it couldn't. We opened trace core to anyone want to extend, which allow to load zipkin core dynamicly in a plugin. You could do that if you want.

[alex-lx]

Sorry, do you mean that the SkyWalking agent supports plugins to add support for other collectors?

[wu-sheng]

Yes, and not just collector, you could get advantage of all SkyWalking agent and plugins, and inject any 3rd party tracing core, and send to anywhere, including Zipkin, file, etc.

We are welcome to this kind of extensions.

[DanielYWoo]

as of 2022, agent is still not supported?

[wu-sheng]

as of 2022, agent is still not supported?

If no one takes action, nothing happens in any oss community. The year doesn't matter.

[DanielYWoo]

as of 2022, agent is still not supported?

If no one takes action, nothing happens in any oss community. The year doesn't matter.

sorry, no offensive, just saying. I used many different APMs but never zipkin. Today, I did a demo with dapr and zipkin is the default APM, then I realized that Zipkin does not have an agent, it's quite something to me.
Just curious, will Zipkin support OpenTelemetry? Then it can take advantage of any existing agents.