Exporting metrics on a default port + path?

[emschwartz]

Should autometrics libraries automatically or optionally start a server that exports metrics on a default port + path?

Prometheus and OpenTelemetry libraries in different languages seem to have a mixed view on this. Some expect you to add the listener yourself while others do it automatically.

Standardizing this across languages would help reduce the configuration needed for any additional tooling we build around autometrics. For example, we could ship a default Prometheus config and/or config files for hosted Prometheus services like Fly.io's.

That said, we might want to make this optional (though not sure if it's opt-in or opt-out). I personally find it surprising that a library I import would open an listener on a port without my explicitly telling it to.

Unfortunately, if we do this, this is another place where OpenTelemetry, OpenMetrics, and Prometheus diverge 🤦.

• OpenTelemetry's Prometheus Exporter lists the port as 9464 (though this is marked as experimental)
• OpenMetrics apparently requested 9099 from IANA
• Many Prometheus clients export metrics on 9091

[gagbo]

It seems like a lot of work, because on top of the OT/OM/Prom default setting variance, there are at least 2 other sources of divergence:

• the difference in http server libraries: just for Rust, you'd have to plug the code within Axum, Warp, Actix-Web, Rocket to add a route that would be exposed, and there are a few languages where that library choice might be wide (on top of implementing the HTTP server manually on top of a raw socket)
• the difference in authentication handling: some deployments might want to add authentication in front of the metric endpoints to avoid scraping (if the monitored API is public for example). In that case, autometrics would also need to provide implementations that work correctly with any middleware the AM client might already have in place

Since there's no real way to handle those differences at runtime (unless the language is very dynamic), that puts a lot of pressure on the code generation/macro part of AM to implement all the combinations for all use cases in my opinion

[emschwartz]

the difference in http server libraries: just for Rust, you'd have to plug the code within Axum, Warp, Actix-Web, Rocket to add a route that would be exposed

I think we wouldn't necessarily need to. This would be a different listener on a separate port, so we could use any library we want (in Rust it would probably be plain hyper). If you're not using that library and are worried about pulling in the extra dependency, you'd be free to not use this function and write it yourself.

the difference in authentication handling: some deployments might want to add authentication in front of the metric endpoints to avoid scraping (if the monitored API is public for example).

I think that's somewhat unlikely. One of the points of listening on a separate port would be that you could expose it within, say, your Kubernetes cluster without exposing it to the public internet.

In that case, autometrics would also need to provide implementations that work correctly with any middleware the AM client might already have in place

I disagree, because this would be a case where we'd say just write this yourself. If you want anything more complicated than a little server listening for HTTP GET requests on a single port + path, we would still provide the functions to gather the metrics into a string and you can implement whatever logic you want with that.

that puts a lot of pressure on the code generation/macro part of AM to implement all the combinations for all use cases

I don't think any of this would happen in generated code. In my mind, these would be helper functions exported by the library that you'd explicitly call in your code (or they'd be included in whatever setup function you need to call initially to start the collection of metrics).

[IvanMerrill]

I agree with you @emschwartz - it seems to make sense to make autometrics provide an easy way to expose the metrics to be scraped. It's another hurdle people would face in order to use the metrics so if there's an easy way to do this that solves the problem for most people then provide it, whilst giving people the option to do something bespoke if they need it.

[emschwartz]

Okay, I think I want to propose using OpenTelemetry's port 9464 by default. That was originally the port used for the OTel JS implementation specifically but they intentionally decided to use that same port for all implementations of the OTel Prometheus exporter (open-telemetry/opentelemetry-specification#1021).

Prometheus has a big list of ports used by different exporters. Theoretically, we could pick a number there. But I think you could make a case that autometrics isn't an exporter in its own right. Rather, it's an opinionated layer on top of OpenTelemetry, so it makes sense to just export the metrics it creates in the normal OpenTelemetry way.

[actualwitch]

My $0.02:
I think starting a separate server from just a library import is indeed weird. Looking at the python prom client for example, it exposes a separate function to do so, and a bunch of request handlers you can plug in your existing server. I think we should do something similar as most devs would expect it. As for port, I think if we want to provide a quick start experience for people just getting into observability, they probably don't have many ports busy with prom accessories, so we can pick the option that causes least confusion - personally, I think 9091 would fit the bill (because it's not far off 9090 and easy to remember). If we expose a function we can also give users an option to use other port instead.

[emschwartz]

Fair point. I agree we can provide some separate functions to do this and maybe just specify it as the recommended way to get started in the docs.

Interesting point on 9091. One argument against that choice in particular is that's listed as the port for the push gateway in their wiki 😕.

[actualwitch]

9095 is also a good option 😄 or like 19090

[gagbo]

I'm fine with picking any port that's not already allocated; that prevents the risk of telling people to copy-paste/reuse a snippet of prometheus.yaml that actually points to an exporter they were already using/planning to use.

[actualwitch]

python library for prometheus exposes a function that starts the server, and it accepts optional parameter for port. i think we can just do the same (ie just reuse the function in case of am-py)

[hatchan]

Another issue with running this on a predefined port is that it will make it make it very tricky for multiple services to be run at the same time, specially if they startup the server by default. We can make it easier for users by providing the handler logic and allowing them to hook it up to their own webserver. The ease of use and quality of this does depend on the language.

Regarding usage with am I think I would opt for a separate approach of just defining a config file that will be read during startup, which contains all the endpoints that you want scraped. In a single service this file would just be contained in the repo, with multiple services you can consider storing this in another repo. If some of the services are not running Prometheus would just try to scrape them, fail, and that would be it.

[emschwartz]

That's a good point about having multiple services running at the same time.

I wonder how likely it is that you'd be running multiple autometricized services locally but not running them in Docker. If you have a microservices setup and you're testing things locally, I think you might be running something like Tilt or Docker Compose and it seems like you might have Prometheus included.

In general, I'd really like to avoid config files as much as possible.

[hatchan]

I wonder how likely it is that you'd be running multiple autometricized services locally but not running them in Docker. If you have a microservices setup and you're testing things locally, I think you might be running something like Tilt or Docker Compose and it seems like you might have Prometheus included.

We ourselves are running our services with Tilt and are running them on the host itself. I startup services as-is as well, and so I don't think it is an uncommon scenario.

[emschwartz]

That's true -- one important thing though is that having a default doesn't preclude allowing the user to configure it. If you have a complicated setup, it makes sense that you'd need to tell us where to get the metrics from. But if you have a simple setup, we have the option to make it very easy for you

[hatchan]

I think the general consensus is that it is a bit odd to have this server startup when including a library, not to mention all the edge cases with the same port being used and even when using different version of the same library in a Rust application.

Library authors are encouraged to make it easier to expose the metrics by encapsulating the logic for various popular web frameworks, but should not start any servers by default.