parse error.log

[63110]

Hi.
I wanted to get log from error.log, but I can't to find format log.
I know that, we can't to change log_format for error.log only level of errors. But which log_format use to parse?
thanks for any suggestions
I read this article #185 but it's not work for me.
Can't see fully messages

namespace "endpoint" {
source = {
files = [
"/var/log/nginx/endpoint.log"
]
}
format = "$date $time [$level] $pid#$tid: *$cid $message"
relabel "level" { from = "level" }
}
result

endpoint_http_response_count_total{level="alert",method="",status=""} 1

[martin-helmich]

How does your error log look like? Does it match the format "$date $time [$level] $pid#$tid: *$cid $message", or does it look like a different format entirely?

[63110]

Yeah, My error.log

2023/02/06 09:08:24 [alert] 105326#105326: *149 shared connection is busy while proxying and sending to client, udp client: 111.120.11.00, server: 0.0.0.0:17, upstream: "00.00.00.94:80", bytes from/to client:15820296/356453044, bytes from/to upstream:356454496/15820296

[martin-helmich]

But, from your metrics, it looks like the errors are actually scraped successfully, and it even got the error level correct (apart from the metric being misleadingly named http_response_count, because that's just hardcoded in the exporter).

If you want to expose the $message as label, you'll need to add a relabeling config. Be advised that this may result in a high metrics churn rate¹, leading in turn to a high number of different metrics and potentially increased memory consumption. In that case, I'd also suggest adding additional capturing groups to your log format to catch all the connection metadata that are present at the end of the line (so that they're not caught by the $message group).

Footnotes

1. https://valyala.medium.com/prometheus-storage-technical-terms-for-humans-4ab4de6c3d48 ↩

[63110]

Ohh. Now I stuck with problem, that delimiter is gap. Which don't allow to parse full message

but message field
*149 shared connection is busy while proxying and sending to client, udp client: 111.120.11.00, server: 0.0.0.0:17, upstream: "00.00.00.94:80", bytes from/to client:15820296/356453044, bytes from/to upstream:356454496/15820296.

P.s I tried to use split and separator. but the same situation

[martin-helmich]

The regular expressions that are generated from the access log format are non-greedy by default. You should (untested!) be able to enforce matching the whole message by adding additional matchers after the $message -- even a generic $remainer matcher at the end should suffice, as long as it is unambiguous:

format = "$date $time [$level] $pid#$tid: *$cid $message, $protocol client: $client, server: $server, upstream: \"$upstream\", $remainder"

Also, on a side node:

• Do not use high-cardinality values (like process/connection IDs, timestamps, and so on) as label values in a Prometheus metric. Prometheus will store all different combination of label values as separate time series, resulting in a massive number of time series, which in turn will degrade performance and increase your memory consumption proportionally.

[63110]

without changes...
does not match given format...
Can I somehow parse it differently for verification?