[[TOC]]
We deploy the pushgateway in a VMs, and we have one for each environment:
gprd:blackbox-01-inf-gprd.c.gitlab-production.internalgstg:blackbox-01-inf-gstg.c.gitlab-staging-1.internalops:blackbox-01-inf-ops.c.gitlab-ops.internal
The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted via the Pushgateway's API.
Imagine a scenario where we want to delete all the metrics for the job="walg-basebackup" that have a type="null".
-
SSH inside of the pushgateway VM, and validate that the metric is there.
$ ssh blackbox-01-inf-gprd.c.gitlab-production.internal [email protected]:~$ curl -s 127.0.0.1:9091/metrics | grep 'null' gitlab_job_failed{instance="",job="walg-basebackup",resource="walg-basebackup",shard="default",tier="db",type="null"} 0 gitlab_job_max_age_seconds{instance="",job="walg-basebackup",resource="walg-basebackup",shard="default",tier="db",type="null"} 108000 gitlab_job_start_timestamp_seconds{instance="",job="walg-basebackup",resource="walg-basebackup",shard="default",tier="db",type="null"} 1.723680043e+09 gitlab_job_success_timestamp_seconds{instance="",job="walg-basebackup",resource="walg-basebackup",shard="default",tier="db",type="null"} 1.723730422e+09 push_failure_time_seconds{instance="",job="walg-basebackup",shard="default",tier="db",type="null"} 0 push_time_seconds{instance="",job="walg-basebackup",shard="default",tier="db",type="null"} 1.723730422461835e+09
-
Delete the metrics:
[email protected]:~$ curl -X DELETE http://127.0.0.1:9091/metrics/job/walg-basebackup/tier/db/shard/default/type/null
Note the URL will require you to have most of the labels, to target the specific metric. For example above we had to specify the
job,tier,shard, andtypelabel. For more information how to construct the label check pushgateway documentation -
Check if metrics are still there:
[email protected]:~$ curl -s 127.0.0.1:9091/metrics | grep 'null' [email protected]:~$
You might also want to validate that it's not longer available on dashboards.