Stats export

[m90]

#60 introduced the collection of stats about a backup run (e.g. size, duration, number of pruned items). These stats can currently be consumed in post-backup notifications, however it should also be possible to send these to arbitrary monitoring systems (e.g. Prometheus) so that users can monitor their backups on such systems.

I don't know too much about the monitoring landscape out there, but in case someone would like to use this feature, I'd be happy to hear from you about how this could work in your setup.

[rpatel3001]

Instrumenting for prometheus should be fairly easy (https://prometheus.io/docs/guides/go-application/). I think most common monitoring systems can ingest the data by scraping the /metrics page.

[pixxon]

Instrumenting the Go application is indeed fairly easy, since Prometheus itself provides the tools, but the issue here comes from the application not running continuously. This can be connected to #268, which would also require the application to take control and be the entrypoint of the containers.

Regarding how Prometheus operates, the microservices expose a metrics endpoint, most commonly /metrics ( and sometimes on a seperate port than where the service itself serves ). This is scraped by Prometheus itself periodically and it stores the information in its database. Grafana can be added on top of this, to visualize the data and create dashboards for easier monitoring.

I would personally try to avoid having a state for a backup application, so the metrics should be limited to information available since last startup. The metrics I would be interested in: sum of backups, percentage of successful backups, how long were the containers stopped, how long it took to complete the backup.

[m90]

I looked into this in https://github.com/offen/docker-volume-backup/tree/serve-metrics once. busybox comes with a simple enough HTTP server that can run if requested and serve a file containing the Prometheus-formatted metrics since last startup.

All it would need is some logic that serializes the already available metrics (plus maybe some others) into the correct format and writes them to a file.

This can be connected to #268, which would also require the application to take control and be the entrypoint of the containers.

That'd also be an option in case busybox's httpd has some limitations I don't know of yet