Feature request: Exportable format to other receivers

[mikkeschiren]

First of all, I have good use of SPX, and have had great experience of using it with various projects. With the rise of OpenTelemetry etc. is it possible to export traces to receivers like zipkin and jaeger? I know the formats from SPX is different, so is it a lot of work to get that into SPX?

[AlexNDRmac]

I already have one converter from SPX full profile to FlameGraph and third-party profiling system for more flexible visualization and profile comparison. It's not so difficult to convert from SPX to other formats. But it depends on input specification from third-party services/app which you'd like to use for profile visualization.

[NoiseByNorthwest]

@mikkeschiren Why do you want SPX "trace" to be processed by monitoring / tracing tools such as zipkin or jaeger ? What use cases do you want to support with that ?

[NoiseByNorthwest]

I already have one converter from SPX full profile to FlameGraph and third-party profiling system for more flexible visualization and profile comparison. It's not so difficult to convert from SPX to other formats. But it depends on input specification from third-party services/app which you'd like to use for profile visualization.

In the past, before the introduction of web UI, SPX supported third party formats such as Google Trace Event or Callgrind. The web UI makes them less relevant now.
However, do you see any format worth supporting ?

[AlexNDRmac]

@NoiseByNorthwest current web-ui is great, it simple as possible and shows everything what needs for app investigation. In my specific case - I need some historical traces with comparison ability, so that is the reason why I wrote converter from SPX format to third-party Application input format. But, will be very useful if SPX be able to export to other formats, like pprof.
This format natively supports by open source continuous profiling system, it might be useful and will be great if SPX profiler can export data to such systems.

[NoiseByNorthwest]

There is already a discussion about pprof here #179

In my understanding it could be a bit complex to support due to a dependency to protobuf. I may be wrong.

[AlexNDRmac]

its not a tricky to use protobuf, but at this time I can't estimate how much time needs for pprof format implementation. Because it depends on external visualization system input protocol. What I mean - various metainformation in trace like units (second, nanosecond... bytes, megabytes, etc...), or full filepath, or other data like inc/exc percentage, call count, etc.

[mikkeschiren]

My thinking is when you need to trace an application in a distributed environment, like in Kubernetes. And collect all the traces to the same endpoint. Personally - I prefer to use SPX and other tools locally, but we are getting asked for to have a common endpoint to collect all applications.
I know this is not the main goal with SPX, I get that.

[AlexNDRmac]

@mikkeschiren why you need to collect all the traces from distributed apps? Is this clause not covered by OpenTracing?
For instance:

• you have many apps which called each other. With OpenTracing (now it OpenTelemetry) you can get whole picture of microservices communication with Jeager UI or other visualization, or with flamegraph visualization.
• You can easily detect which microservice has an issue and you need to check it with profiler, so, you don't need to collect all the traces from all the microservices - it's very expensive, especially if you collect many metrics (cpu, memory, allocations, gc, etc...)
• Next - you can check this standalone microservice with SPX, foud and fix some issue and recheck whole system.

If you want to profile every microservice in the real time - it's called continuous profiling and with SPX it can be difficult.

[mikkeschiren]

Yes I am aware this - though auto tracing is not yet possible with OpenTelemetry (what I know). The requests itself comes from some of our developers.

[AlexNDRmac]

Does it stop developers from own implementation? :) Nothing stops you from using SPX or other tool to collect profiles from PHP app, convert profiles and sent to any of external system for profiles analysis. What might be useful - is to add to SPX ability to get some information from request, like correlation_id or request_id, etc... and pass it as metainformation with profiles. It'll helps to sort out with a lot of profiles in external system and found traces by request ID.

[NoiseByNorthwest]

@AlexNDRmac I think that the ability to specify tags, described here #173 (comment) would cover that.

[AlexNDRmac]

@NoiseByNorthwest - yes, Tags will cover all the metainformation or other custom data!

[AlexNDRmac]

@NoiseByNorthwest, a few days ago Grafana introduced Phlare. It's a database for Continuous profiling.

What do you think about pprof format implementation for SPX profiler? If SPX will be able to write to pprof format - it'll be very interesting solution with Grafana Phlare.

[NoiseByNorthwest]

To be honnest I don't know, and from the beginning I had never envisioned that SPX could be a production tool.

But I'm open to the idea, I just need a quite detailed specification (bunch of bullet points is enough) about the pprof format integration and also how SPX would push profiles to Phlare.

[mikkeschiren]

When asking in phlare issue queue to post pprof to phlare the answer was:
$ curl -v -H "Content-Type: application/json" -d '{"series":[{"labels":[{"name":"__name__","value":"profile_cpu"},{"name":"job","value":"manual"}], "samples": [{"raw_profile": "'$(cat profile |base64 -w 0)'"}]}]}' http://localhost:4100/push.v1.PusherService/Push

There is also an open issue on using the phlare cli for uploads:
grafana/pyroscope#2088

[NoiseByNorthwest]

Wouldn't such feature be far easier to implement in Go ?
I mean it could be a simple service watching SPX data directory and converting on the fly SPX format to pprof and then sending it to Phlare.

[BafS]

Could it be possible to re-introduce callgrind format? The web UI is great but it's sometimes complex to setup (for example because of CSP issues with pre-prod, because we don't have internet and jquery is taken from the web, etc.) or because we profile something via SSH and we don't have an UI. Having a standard format would alleviate those issues.

[NoiseByNorthwest]

Could it be possible to re-introduce callgrind format? The web UI is great but it's sometimes complex to setup (for example because of CSP issues with pre-prod, because we don't have internet and jquery is taken from the web, etc.) or because we profile something via SSH and we don't have an UI. Having a standard format would alleviate those issues.

This is possible but I dont know when I'll have the time to do that. On the other hand the "full" report could be converted to callgrind format with a <50LOC PHP script.

[BafS]

Okay, thanks for your answer! Is there any documentation about the SPX format? I'll have a look when I get some time

[NoiseByNorthwest]

Take this example:
metadata: https://github.com/NoiseByNorthwest/NoiseByNorthwest.github.io/blob/master/php-spx/demo/data/reports/metadata/spx-full-20191229_175636-06d2fe5ee423-3795-233665123
data: https://raw.githubusercontent.com/NoiseByNorthwest/NoiseByNorthwest.github.io/master/php-spx/demo/data/reports/get/spx-full-20191229_175636-06d2fe5ee423-3795-233665123

The metadata file is a self-describing JSON string.

The data file is a text file, designed to be read line by line, and containing 2 sections.

The first section is the "events" section, starting with [events]. Each line in this section follows this format:
<function index> <1 if start of function call, 0 if end of function call> <metric #1> <metric #2> ...

The function index is a reference to the corresponding line in the second, "functions", section.
The metrics are the ones referenced (same order) by the "enabled_metrics" property of metadata.

The second section is the "functions" section, starting with [functions]. Each line in this section is a function name. The relative line number (i.e. starting from 0) is the function index referenced in the first section.

[NoiseByNorthwest]

Regarding metric values:

• time-related metrics are integers representing the relative time (from start of script execution) in nanoseconds
• memory-related metrics are integers representing bytes

Additional information for each metric here: https://raw.githubusercontent.com/NoiseByNorthwest/NoiseByNorthwest.github.io/master/php-spx/demo/data/metrics

[BafS]

Thank you for all those info @NoiseByNorthwest!

I finally did a basic script to convert it in trace-event format: https://gist.github.com/BafS/b677f0f292a00c3aae75d20df55e4f1a