Autometrics in Elixir

[cognivore]

Hi, @emschwartz! We chatted at the conference, and on the next day I tried to guess your E-Mail address, but failed.

I'm tempted to write a small library based on this for injecting code, but for the time being, here's a sketch of a naive redefined defmodule (! you have to redefine defmodule because def doesn't have access to @doc !) for your atometrics:

Given:

iex(1)> quote do
...(1)> defmodule T
...(1)> @doc "hello"
...(1)> def f(x), do: x + 100
...(1)> end
{:__block__, [],
 [
   {:defmodule, [], [{:__aliases__, [alias: false], [:T]}]},
   {:@, [context: Elixir, import: Kernel],
    [{:doc, [context: Elixir], ["hello"]}]},
   {:def, [context: Elixir, import: Kernel],
    [
      {:f, [context: Elixir], [{:x, [], Elixir}]},
      [do: {:+, [context: Elixir, import: Kernel], [{:x, [], Elixir}, 100]}]
    ]}
 ]}

Redefine defmodule as:

defmacro defmodule(alias, do_block) do
  [h | fs] = super(call, expr)
  # Iterate over fs, tracking calls to `@`, the first argument of which is `:doc`
  # The next function head will be the one for which this documentation is written
  # Do the same by tracking calls to `@spec`.
  # The next function head will be the one for which the typespec is written (detect
  # {:ok, t()} | :error here with a bunch of heuristics).
  # Eventually, insert metric_start and metric_end into the do_block of that function as you need it.
  # Now that you know the function name, and if it's "result", you can append to `@doc` and
  # replace it too.
end

This will mess up other macros that redefine defmodule (lol), so you should check out Qqwy's sketch to prevent unsafe overrides: https://gist.github.com/Qqwy/d7d3fe9e172edc9a2a836f6754375819

Sorry for not writing the actual code, but these bits should be good enough for copilot to write the macro for you haha.

Hope it'll be a breeze to release autometrics for Elixir... I'm absolutely gonna learn how to run prometheus and grafana on a plain VPS to test it in production!

P.S.
I found the error data types article too:
https://www.shuttle.rs/blog/2022/06/30/error-handling

P.P.S.
Are there any forums you hang out on (aside from tw*tter), would be cool to stay in touch!

P.P.P.S.
Correct link to the article: http://sled.rs/errors.html

[emschwartz]

Hey @cognivore, thanks for following up! It was great meeting you at the conference :)

We actually had two people, @actualwitch and @stephlow, start poking around trying to implement it in Elixir yesterday so your suggestion came at a perfect time. I'll let them follow up with ideas/questions, since they're more in the weeds with trying to implement it.

I'm not a big forum person and am embarrassed to say I'm still on the blue bird site ;) but if you want to come chat about autometrics, we've got a dedicated channel on Fiberplane's Discord https://discord.gg/kHtwcH8As9

[cognivore]

This is an embarrassing failure of autocutsel. Only now I realised that I pasted a link to the wrong article in the OP!!!

http://sled.rs/errors.html <- here's the correct one @emschwartz

[emschwartz]

Thanks @cognivore for the initial ideas around this! @actualwitch and @stephlow played around with a couple of things and it seems like there are a couple of different ways to implement autometrics in Elixir. None of us are Elixir experts so we could very much use input on these options!

---

To make autometrics work in Elixir, we need two components:

1. A macro that instruments functions with code to track metrics for that function
2. A way to modify the IDE doc comment tooltips to include links to the live charts for each function

Instrumenting Functions

As far as we can tell, there are a number of options that seem viable. The main question probably comes down to Elixir library API design: which of these options would Elixir developers find most natural to use?

Option 1: Overriding def or defmodule macro

One potential option is to override these macros such that the code they output includes autometrics instrumentation.

We would probably want to use the SafeOverride technique @cognivore mentioned to ensure that this override does not conflict with anything else that might be overriding those macros.

As far as we understand, defmodule has the ability to modify the function docs while def does not.

• If we need a separate language server plugin (see below)) to have the chart links show up in your IDE, it might not be necessary to actually modify the function docs
• The only reason you might want to change the actual docs would be if you want the autometrics links to show up in the documentation generated by ExDoc

If we override defmodule, you would need to wrap your module in a higher-order module:

defmodule Outer
  use Autometrics

  defmodule Inner
    def function() do
    end
  end
end

Option 2: defa macro

Instead of overriding system macros, we could define another macro that extends the def macro and adds instrumentation.

To be honest, we are not sure about the merits or drawbacks of this approach.

Option 3: Decorator package

It seems like we could use https://github.com/arjan/decorator. The API would be something like:

defmodule MyApp.APIController
  use Autometrics

  @decorate_all autometrics()

  def index(_conn, params) do
    # ...
  end
end

You could also decorate individual functions:

defmodule MyApp.APIController
  use Autometrics

  @decorate autometrics()
  def index(_conn, params) do
    # ...
  end
end

This is the approach taken by AppSignal’s Elixir library.

• We heard from someone that the performance overhead of this particular library is significant. However, this seems more likely to be due to the actual instrumentation code and logic their library is adding rather than anything to do with the decorator package

Option 4: autometrics macro

It seems like we could define an autometrics macro that would wrap any number of functions:

defmodule MyApp.APIController
  use Autometrics

	autometrics do
		def index(_conn, params) do
	    # ...
	  end
	end
end

We are also unsure about the merits or downsides of this approach. One minor annoyance would be that when you add autometrics you diff would be massive because you’d change the indentation of loads of code.

Showing Prometheus chart links

The Elixir language server / elixir_sense have very limited understanding of macro-generated code. This means that if we modify function documentation in macros, it will not show up when you hover over a function’s name in your IDE.

elixir_sense has a plugin system and according to this comment, they might be open to accepting a plugin for autometrics:

you are hitting a limitation in elixir_sense. It has only a limited understanding of macro generated code. In order to have first class support for Absinthe it would require a dedicated plugin. We already have one for Ecto (https://github.com/elixir-lsp/elixir_sense/tree/master/lib/elixir_sense/plugins) and we are open for contributions.

The main open questions here are:

1. Would they accept a contribution for an autometrics plugin? Autometrics is a brand new library, unlike Absinthe and Ecto, which are very popular packages
2. Are plugins enabled by default? It’s not obvious whether you need to configure the Elixir language server to use them or if they’re automatically enabled
3. The IDE extensions may be using outdated versions of elixir_sense (see this issue). If so, how long will it take to get our plugin into the VS Code extension or others?

[DianaOlympos]

Trying to catch up on what autometrics is, but let me be clear.

None of these options are acceptable from an elixir dev point of view.
Option 1 is a total no go
Option 2 is a maybe but honestly it means we will quickly stop using it
Option 3 is the same. This decorator behavior is actively fought by the community
Option 4 is a possibility, but it is also highly possible that people will just stop using it if that is the case.

A better solution, to get the data at least, would be to use dynamic tracing. This is what I do in Orion. Hell tbh, what I do in Orion is exactly targeting your use case, except I do not need Prometheus. I already have plans to extend it to the IDE so you can get your graph directly in your IDE, with no need for any kind of storage.

If you go this way, please consider using dynamic tracing. This way, you do not need to annotate anything or modify the AST.

On the VSCode side. You probably do not need to use it, or more precisely, you probably want to use one of the more recent ways to programmatically modify elixir code. There are a few libraries that are starting to pop up, usually based on sourceror. I do not think they are integrated yet into the elixir_ls repo, but that would probably be a first step, at least if you want to inject something into the comments.

On the old version of ElixirSense, I just checked and the version rn in the main branch is from 11th February 23. Slightly old, but not that much. If you ask for it to be merged or offer a PR, it will probably happen fast-ish.

Feel free to ask more questions as much as you need :)

[emschwartz]

Thanks a lot for weighing in!

Orion looks very neat -- thanks also for the tip about dynamic tracing. If I'm understanding correctly how Orion works, it looks like you've got the TracerSupervisor, which is linked to every(?) process that starts, and then these methods track when a process starts, get the duration when it ends, and add it to the graph. Then it seems like you've got the graph just stored in memory using dog_sketch and served to the Web UI (or in the future the IDE extension), right?

Two questions about this approach:

• How is the time range configured? Or will it always show you the full time range since your application started?
• With the supervisor method, is it possible to exclude functions from its purview if you had some functions or modules that you really didn't care about?

I agree that we're thinking about somewhat similar use cases. Presumably you'd also be able to track and graph the request rate and error rate using this method as well (I'm guessing you'd have a dog_sketch for each one and you'd use whether the process crashed as an indication of it "erroring").

A couple of potential benefits I could see of storing the metrics in an external system like Prometheus would be:

• Correlating metrics about the elixir application with infrastructure metrics
• If you have multiple applications written in different programming languages, having one unified place to look at all of their metrics
• Storing the time series on disk so that you can dynamically adjust the time range you're looking at (maybe you have plans for on-disk storage? Though I'm curious if dog_sketch lets you adjust the time range)

Lastly, thanks also for the tip about sourceror!

[DianaOlympos]

so multiple things.

1. I generally create one process per trace, but in theory, you do not have to. Up to you. It works nicely for me because I allow dynamically "stopping" a trace at any point, so I can just kill the process and be sure there is nothing left. Free GC.
2. The time range right now is that I create one ddsketch per second. And I keep 1min. In theory, you could do whatever you want :D It works for me because they are easy to merge and small on the wire. If you do a prom thing, you probably do not need that but a simpler set of metrics. Otel could work.
3. I mean, I use tracing. So it basically traces nothing except what you want. The main advantage is that the user can define on the fly what they want and only while they have something to look at. So yeah, not storage of the past, but also no need to care about any overhead and no need to store. You get the metrics you want when you need them, nothing else, no need to think about it when you do not need it. Which is different from what you offer. You could use dynamic tracing too, it is not that different from ebpf, to hook into it. And just always trace.
4. Request rate: yes, it is just a count on the histogram. Error rate: no, right now I do not handle that. Tbf, in general, if it errors you know it in Elixir. Here I am looking at providing performance, not APM.
5. Correlating: usually, the problem comes the other way around. You come to Orion and this kind of performance problem because you already know something went wrong. What Orion allows you is to have the ability to go deeper. In my experience, this kind of "shallow" infra metrics is basically useless for performance exploration, except when they are glaringly obvious. So correlating is not that useful. In nearly all cases I have to deal with a performance problem that needs function-level performance, I need to explore the system. Pre-recorded information is useless and usually not granular enough in terms of input (does it happen only when the input is a specific user) or in terms of time precision.
6. For different applications in different programming languages, sure indeed :) As I was saying, feel free to copy it for your work :)
7. Dog_sketch is one per unit of time. Histogram compression over time is a really complex problem that has no good solution afaict. And I try to keep tabs in that domain :D So adjusting the time range is done by keeping as many as you want. So yes, storing to disk or shipping to Prometheus could make sense if you want history. I have never found it useful, and damn knows I built Orion because it solves problems I have. But if it is your goal, feel free to go do it. That said keep in mind that Elixir systems tend to be.... optimized. Storing a ton of metrics at the function level is... not really that useful. In my experience.

But eh, that is what your whole thing is, so do not listen to me being a downer. Go experiment :)

[Nickforall]

My two cents here:

Largely agree with @DianaOlympos here. None of these options are super great.

Implementation

Like @DianaOlympos said, option 1 and 2 are unacceptable, unlikely to be adopted by the Elixir community and probably break a lot of tooling/syntax highlighting. I'd refrain from these anytime

Option 3 is not loved by the entire community, but is definitely the least breaking. Afaik I made comments about appsignal performance to mies, but that was largely caused by their proprietary daemon.

Option 4 is probably the most idiomatic elixir, but I do share your concerns about it impacting readability and causing stupidly large diffs, so does not sound like a great solution to your problem. (I probably would not use it because of readability concerns).

TLDR, I'd pick option 3. It's not idiomatic, but it works!

To be quite honest: I'd use telemetry as much as possible to learn about the time certain operations take. It's implemented by most libraries (ecto, absinthe, phoenix) and gives ample information. There are many different backends and integrations for telemetry, which makes me wonder whether you really need to build yet another tool for this.

Showing chart links

Rewriting docs to include links is tough. Every tool relies on Code.fetch_docs/1 here. These docs are stored in predefined chunks in bytecode, and afaict there's not really a way to modify the AST at an early enough stage without hacking defmodule (option 1).

You may be able to pass some options or write a custom formatter for ex_doc to add it to the html of generated docs. That's the best I can come up with right now.

Or roll your own vscode plugin, which shows gitlens-like annotations.

[maartenvanvliet]

Agreed with @Nickforall here.

Note that while AppSignal offers the decorator approach, it's always been too magical for me. In past projects I never used it and instead hooked up the phoenix/ecto/etc telemetry events to their Transaction api.

[DianaOlympos]

Sourcerror and the newer elixir have ways to modify the code. It is an area of active work

[actualwitch]

Thank you all for great pointers! I've been looking into options mentioned here — sourceror in particular. While it does provide some cool functionality and zipper struct that seems extremely useful, it does seem to me that it is targeting a slightly different use case than ours. In particular, all the functions seem to work on the text source that you already have and it doesn't cover the dynamic patching part which is what I'm currently struggling with. Apart from overriding defmodule (which doesn't seem to work anyway and happens before compilation, if I understand correctly), I see only one option to monkey patch doc attributes - do the same thing that decorator package does, and use with before_compile hooks. I'm not sure if that would work, especially with before_compile only being able to insert code at the bottom of the module. If that doesn't work, it seem that the only option we're left with is a separate extension.
Other thing, I'm not sure instrumenting everything by default is a good idea, because it can generate a whole lot of noise, especially on larger apps. So for a more robust solution I think it would be best to at least offer an option to instrument on per-function basis, and ideally it should just work directly with either erlang's trace or elixir's :telemetry. For that to happen we need to be able to somehow mark functions that we care about, and retrieve the marked function signatures when starting autometrics, but what would be the best option for that? Would using decorators here just to collect the function signatures be acceptable?

[DianaOlympos]

honestly, if that is the solution you end up with, I would consider recommending against this in any codebase I touch. But I understand where you are coming from. I am not sure there is a good solution to this set of constraints and goals. It may be the best you can do.