What encoding?

[Fristi]

I was at your talk today at Fiberplane, great place and product your building!

I've been working with Prometheus/Grafana as well to measure, visualize and do alerting. It does the job, but my main problems with this system is:

• You measure in your application (Prometheus)
• You define dashboards in Grafana (end result: JSON)
• You define alerts in Grafana (end result: JSON)

The end result is a system which requires you to implement this in your code and also some parts in a declaretive style in JSON which is composable in a sense (but not as much a programming language could) and it's not easy to test (hence alerts are not easy to test). The end result is a non-cohorent system in my opinion which is also hard to maintain, since you have to store the JSON grafana models somewhere (git?), synchronize them maybe between different environments and such. Also there is only support from libraries to export pre-configured dashboards or alerts.

I think autometrics makes metrics simpler, but it's only bounded to a counter now right? It also defines the concept of a SLO in code which is awesome, but it sounds like you are building a sort of language (with annotations) and alert constructs in code which could maybe form it's own language?

I was brainstorming my self about a separate DSL which allows you to

• Define metric type like counters, gauges, histograms and such
• Define visualizations like rate/per minute, failures and other visualiizations which are pretty standard and make sense. These visualizations would reference the metrics in the DSL
• Define alerts via ??? .. since Grafana use PromQL to gain an aggregated view and applies window rules to trigger an alert eventually maybe the ??? is a missing abstraction layer.
• By using the DSL you can export a prometheus/otel metric, but also a Grafana dashboard/alert or any other visualization/alerting platform which supports the encoding. But maybe if this it is too complex or constraining to export, a new Prometheus/Grafana solution which solves this a whole might a good idea
• A library/framework could choose to use this standard to pre-package well defined visualizations, alerts and metrics for example http servers, connection pooling (db/http), http clients, etc

[emschwartz]

Hi @Fristi, thanks for sharing this idea and I'm glad you enjoyed the talk!

Just to clarify the autometrics side of things:

• Right now we're focused on metrics that are linked to functions, specifically the request rate, error rate, latency, and optionally concurrency (that uses a counter for request and error rate, a histogram for latency, and a gauge for concurrency). The hypothesis is that this set of metrics gets you pretty far towards measuring what you care about, without having to spend any time thinking about defining metrics. (You can read more about this line of thinking in The Case for Function-Level Metrics)
• We have an open thread of discussion regarding Support for custom metrics, but that will likely be limited to counters unless we can come up with specific use cases that require other metric types
• You're right that the way we define SLOs in code is a kind of small language for expressing SLOs. That said, it's a very small language cause there are only two types of SLOs that we support at the moment. A big part of the hypothesis here is that it's a much nicer experience for developers to define things in the same language they're programming the rest of their system in.

Regarding the observability DSL idea you're describing:

• I think I understand why you're thinking along these lines, because it does seem slightly odd for some bits of the whole observability lifecycle to be expressed declaratively in config languages and some parts to be in code
• I'm having a bit of a hard time imagining how the DSL would be used by the code to produce the data it needs, but maybe there's some way to do that?
• Overall, this feels like a very hard thing to express in a completely generic way. That's kind of why with autometrics we're focusing on a fairly opinionated set of metrics and then giving you all the related visualization, alerting, and testing tools we think you'd want
• How would you describe the benefits of all this being a separate language?

[Fristi]

Hmm not the best idea maybe, going over your last three points and all the responses are a bit wacky as well :) E.g. generating code/dashboards/alert from the DSL. Same idea as infrastructure as code, where the declarations are translated to imperative statements (but in this case there is no real convergence to a certain state)

Coming back to your point

I think I understand why you're thinking along these lines, because it does seem slightly odd for some bits of the whole observability lifecycle to be expressed declaratively in config languages and some parts to be in code

Exactly this!

I think the experience regarding creating visualizations and alerts could be improved. This might not be the scope of this project (yet), but it's maybe some to consider as well.

This is where the idea of one DSL came from actually, one solution where everything references to the same thing and is constrained in a particular manner. However it might not be the best means to achieve this goal, maybe there other ways?

Food for thought, seed is planted 🌱 the idea might come later 🧠