Response for /metrics Endpoint is getting really big over the time

[pandyamitesh]

We have setup the prometheus-nginxlog-exporter and promethus is scrapping the metrics endpoint at every 1 minute. The data provided by /metrics are getting really big over the time , is there a way to limit this ?

[martin-helmich]

If your dataset grows continuously, this might be an indication of a misconfiguration regarding your metric labels (another user had a similar issue not too long ago; see #199 (comment) for more information).

Without seeing your actual configuration, this is all guesswork, however. 🤷

[pandyamitesh]

Log Configuration

log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 'httpstatus:$status $body_bytes_sent "$http_referer" "$http_user_agent" ' '"$http_x_forwarded_for" "$request_id" "responsetime:$upstream_response_time" ';

prometheus-nginxlog-exporter.hcl config

`
listen {
port = 4040
}
namespace "nginx" {
source = {
files = [
"/var/log/nginx/access.log"
]
}
format = "$remote_addr - $remote_user [$time_local] "$request" httpstatus:$status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_id" "responsetime:$upstream_response_time""

labels {
app = "dev-nginx"
}
relabel "request_uri" {
from = "request"
split = 2

match "^(/[^?]{0,})*" {
  replacement = "$1"
}

}

}`

[martin-helmich]

relabel "request_uri" {
  from = "request"
  split = 2

  match "^(/[^?]{0,})*" {
    replacement = "$1"
  }
}

This regular expression matches the entire path of the URL, doesn't it? This will create a new label value for each possible URL, and a new time series with it.

I would strongly discourage defining labels with an (potentially) unbound set of values, as each observed combination of label values will result in a new time series (that's just how Prometheus works). And a potentially unlimited amount of time series also means potentially unlimited memory requirements.