Permission Denied Issue When Deploying Prometheus Nginx Log Exporter as a Systemd Service

[susson]

Hello author,

I'm a newcomer to operations, and I've encountered some issues while using your software. I'm eager to receive your guidance.

Firstly, let me explain my use case:
I have configured syslog on an Nginx server, which sends logs to an exporter. This exporter then provides the data to Prometheus. The exporter is deployed on a CentOS 7 server. When I start the service using the command "prometheus-nginxlog-exporter -config-file /path/to/config.hcl", the service starts successfully, and Prometheus successfully retrieves the data.

Next, I wanted to deploy the exporter as a systemd service, and this is where I encountered an issue. When I check the system's message log, I can see the following content:

Aug 21 22:42:54 server-docit systemd: Started NGINX metrics exporter for Prometheus.
Aug 21 22:42:54 server-docit prometheus-nginxlog-exporter: 2023-08-21T22:42:54.958+0800#011info#011prometheus-nginxlog-exporter/main.go:162#011loading configuration file /etc/prometheus-nginxlog-exporter.hcl
Aug 21 22:42:54 server-docit prometheus-nginxlog-exporter: 2023-08-21T22:42:54.959+0800#011info#011prometheus-nginxlog-exporter/main.go:139#011starting listener for namespace nginx
Aug 21 22:42:54 server-docit prometheus-nginxlog-exporter: 2023-08-21T22:42:54.959+0800#011info#011prometheus-nginxlog-exporter/main.go:148#011running HTTP server on address 0.0.0.0:4040, serving metrics at /metrics
Aug 21 22:42:54 server-docit prometheus-nginxlog-exporter: 2023-08-21T22:42:54.960+0800#011info#011prometheus-nginxlog-exporter/main.go:222#011running Syslog server on address udp://192.168.100.114:514
Aug 21 22:42:54 server-docit prometheus-nginxlog-exporter: panic: listen udp 192.168.100.114:514: bind: permission denied

This seems to suggest a permission issue, but I should have already set the permissions to root. Here is my "prometheus-nginxlog-exporter.service" file:

[Unit]
Description=Prometheus Nginx Log Exporter

[Service]
User=root
Group=root
ExecStart=prometheus-nginxlog-exporter -config-file config.hcl
WorkingDirectory=/etc/prometheus-nginxlog-exporter
#StandardOutput=file:/etc/prometheus-nginxlog-exporter/output.log
#StandardError=file:/etc/prometheus-nginxlog-exporter/error.log
#SyslogIdentifier=prometheus-nginxlog-exporter

[Install]
WantedBy=default.target

Could you please help me identify the issue? Thank you for your assistance!

[asergunovkl]

Having the same issue on one of 5 machines. on Ubuntu. I've made the user not the root but www-data - the same as nginx one. It helped.

[HectorB-2020]

Try removing CapabilityBoundingSet from that systemd unit.
I dropped that line while added other isolation techniques. Works like a charm.

PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=full

[jsabater]

I had the same issue on Debian 12 Bookworm. The file /lib/systemd/system/prometheus-nginxlog-exporter.service runs the exporter as root user but has the CapabilityBoundingSet= directive, which removes all capabilities.

Personally, I added the existing user prometheus (node-exporter was already running on that LXC) to the adm group (since all my logs inside /var/log/nginx/ are owned by the nginx user and the adm group) and modified the file /lib/systemd/system/prometheus-nginxlog-exporter.service as follows:

[Service]
User=prometheus
# CapabilityBoundingSet=