-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus PushGateway publisher logic #325
Comments
Hi @jdomagm thanks for the report. |
@cdsap |
@cdsap any update on this? |
A while ago I had also plans to use Talaiot with Prometheus as well, and noticed that the publisher could be improved. Below is a draft of how I would try to improve the situation: Change descriptionModify what data is exported from Talaiot by default when used with Prometheus Push Gateway publisher. Reasons for change:
Assuming common use case is for collecting metrics for a single project (application), and build executed on developer machines or build server. Build MetricsCommon labelsLabels added to all
MetricsBuild metrics and extra labels if different than common
New Task MetricsMetrics
Reference |
@jdomag With above changes, I think your metric would be possible to extract with |
@apolatynski Thank you very much. I would appreciate it if you can take a look. |
@apolatynski @cdsap One question - wouldn't be better to exclude a task name from the metric name as following:
That would allow people to gather metrics per project e.g. duration of all the tasks within a project |
@jdomag With the initial approach, I believe you can get a metric that you want with a bit ugly query
Mentioned scenario probably would still be handled by Prometheus, but I'm worried that larger projects might already reach 10s of millions of timeseries generated. What do you think @cdsap ? |
very interesting, after reading again the recommendation from Prometheus is clear that using tasks as labels reach the recommended cardinality value(100). However, analyzing our project, we have 600 modules and 13k unique tasks for the main development tasks. The number of modules/projects is a variable that tends to grow and will increase the cardinality of the metric.
I could add one additional property in the
Let me know what you think. Btw I merged the first approach and is available under snapshot in case you want to test it:
|
@apolatynski @jdomag , I finished the update of the Publisher configuration: https://github.com/cdsap/Talaiot/pull/337/files
But when I'm collecting more than one task I get:
Wrongly I assumed I could request unique metrics with different task names as labels. And now I get @apolatynski concern because you must to inform all the tasks on the same metric. I'm not sure if Prometheus offers one type for our requirement:
|
Hi, version 1.5.2 has been published. Please, can you take a look at the pushgateway plugin before we make the announcement? Thanks |
@cdsap
and then configuration looks like this
But I can't see any relevant metrics in push gateway |
I found it great idea to push task metrics to Prometheus and visualize it in Grafana. However current approach is that pushgateway publisher publish each metric with separate name e.g. the name of the metric is set to :project:task. I have 100 projects that execute the same task so metrics in prometheus looks like this:
How am I suppose to create a grafana dashboard showing how much time the task for each project took? I think the metric name should be set on the same value e.g. set by parameter taskJobName in pushgateway publisher. Inside the metric we should see the field "task" or "project"
Then I grab that metric and do something like
gradleTaskDuration{project="xyz", task"abc"}.
Unless there are some other way to use those metrics which I'm not aware of.
The text was updated successfully, but these errors were encountered: