Monitor Podman using unix socket #24103

diasdmhub · 2024-09-29T12:42:50Z

diasdmhub
Sep 29, 2024

I'm trying to monitor my Podman Pods with Zabbix 7, but it keeps failing with the following error message.

Cannot fetch data: Get "http://1.28/info": dial unix /run/user/1004/podman/podman.sock: connect: permission denied.

The Pod is running as a rootless user (UID 1004) with linger enabled.

$ loginctl list-users
 UID USER     LINGER STATE    
1004 user     yes    lingering

Also, I've enabled the Podman service for this user.

$ systemctl --user status podman.service
● podman.service - Podman API Service
     Loaded: loaded (/usr/lib/systemd/user/podman.service; enabled; preset: disabled)
     Active: active (running) since Sun 2024-09-29 08:38:06 -03; 2s ago
TriggeredBy: ● podman.socket
       Docs: man:podman-system-service(1)
   Main PID: 5317 (podman)
      Tasks: 7 (limit: 190751)
     Memory: 14.4M
        CPU: 25ms
     CGroup: /user.slice/user-1004.slice/[email protected]/app.slice/podman.service
             └─5317 /usr/bin/podman --log-level=info system service

And the Podman socket is in listening state.

$ systemctl --user status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; disabled; preset: disabled)
     Active: active (listening) since Sat 2024-09-28 18:38:28 -03; 13h ago
      Until: Sat 2024-09-28 18:38:28 -03; 13h ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1004/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1004.slice/[email protected]/app.slice/podman.socket

With the local user, I can successfully test the socket response using cURL.

$ curl --unix-socket /run/user/${UID}/podman/podman.sock http://d/v5.0.0/libpod/info
{"host":{"arch":"amd64","buildahVersion":"1.33.8","cgroupManager":"systemd","cgroupVersion":"v2","cgroupControllers":["memory","pids"],
(...)
"OsArch":"linux/amd64","Os":"linux"}}

With all that done, I then added the zabbix user to the local user group.

# usermod -aG user zabbix

# groups zabbix
zabbix : zabbix user

$ ls -lah /run/user/1004/podman/podman.sock 
srw-rw---- 1 user user 0 Sep 28 18:38 /run/user/1004/podman/podman.sock

Still, Zabbix requests are denied.

When I try a local request with another user, it is also denied.

$ curl -v --unix-socket /run/user/1004/podman/podman.sock http://d/v5.0.0/libpod/info
*   Trying /run/user/1004/podman/podman.sock:0...
* Immediate connect fail for /run/user/1004/podman/podman.sock: Permission denied
* Closing connection 0
curl: (7) Couldn't connect to server

With Docker, we can create a Docker group and add the necessary users.
However, I'm not sure how to allow the Zabbix user to access the Podman socket.
Any thoughts on this?

Answered by diasdmhub

Oct 3, 2024

What I'm trying to accomplish is to monitor the rootless Pod with Zabbix.

So far, what I'm catching up is that SystemD creates the /run/user directory as root.

[root@pod ~]# /usr/lib/tmpfiles.d/systemd.conf
d /run/user 0755 root root -

In it, if user linger is enabled, a directory is created for the user with permission 0700, which means no group access.

[user@pod ~]$ ls -lad /run/user/1004
drwx------ 8 user user 200 Oct  3 16:17 /run/user/1004

Since the /run/user/$UID directory is created by enabling user linger, I've created the .config/user-tmpfiles.d/podman_user_1004.conf file with the following value. (Thanks @afbjorklund)

z /run/user/%U 0710 user user -

With this, I aim to give …

View full answer

rhatdan · 2024-09-29T12:58:17Z

rhatdan
Sep 29, 2024
Maintainer

You would have to set the socket to group user access of podman with R/W

Start the service
chown 1004:user /run/user/1004/podman/podman.sock
chmod 660 /run/user/1004/podman/podman.sock

4 replies

rhatdan Sep 29, 2024
Maintainer

If you want to make this permanent, you need to modify the podman.sock unit file.

diasdmhub Sep 29, 2024
Author

Thanks for your quick reply.

I think the socket is already set up that way.

$ ls -lah /run/user/1004/podman/podman.sock
srw-rw---- 1 user user 0 Sep 28 18:38 /run/user/1004/podman/podman.sock

rhatdan Sep 29, 2024
Maintainer

Then make sure the other user is actually in the user group. And not being run inside of a container, then it would work.
As other user
groups

Nothing Podman is doing to prevent access.

If you are running another container as a different user and trying to access the socket, then you have other concerns.

diasdmhub Sep 29, 2024
Author

The error persists when a second local user is used (not from another container), and this second user also has access to the pod user group.

[user2@pod ~]$ groups user2
user2 : user2 user

[user2@pod ~]$ curl -v --unix-socket /run/user/1004/podman/podman.sock http://d/v5.0.0/libpod/info
*   Trying /run/user/1004/podman/podman.sock:0...
* Immediate connect fail for /run/user/1004/podman/podman.sock: Permission denied
* Closing connection 0
curl: (7) Couldn't connect to server

diasdmhub · 2024-09-29T13:28:37Z

diasdmhub
Sep 29, 2024
Author

Even when I set the pod user socket to full access 777, it doesn't respond to other user requests.

[user@pod ~]$ chmod 777 /run/user/1004/podman/podman.sock

[user@pod ~]$ ls -lah /run/user/1004/podman/podman.sock
srwxrwxrwx 1 user user 0 Sep 28 18:38 /run/user/1004/podman/podman.sock

[user2@pod ~]$ curl -v --unix-socket /run/user/1004/podman/podman.sock http://d/v5.0.0/libpod/info
*   Trying /run/user/1004/podman/podman.sock:0...
* Immediate connect fail for /run/user/1004/podman/podman.sock: Permission denied
* Closing connection 0
curl: (7) Couldn't connect to server

2 replies

rhatdan Sep 29, 2024
Maintainer

No idea what is blocking this. Try with just a random socket in one user account and attach from another using ncat.

eriksjolund Sep 29, 2024

Maybe the directory /run/user/1004 only allows access to the user user?

On my Fedora CoreOS computer it looks like this

$ ls -ld /run/user/*
drwx------. 7 core  core  160 Sep 28 06:12 /run/user/1000
drwx------. 7 test1 test1 160 Sep 28 23:43 /run/user/1001

(This is the default permissions because I haven't changed anything)

An alternative could also be to change the path of the unix socket

A sketch:

$ mkdir -p ~/.config/systemd/user
$ systemctl --user edit podman.socket
$ cat .config/systemd/user/podman.socket.d/override.conf 
ListenStream=/some/other/path/podman.sock
$ systemctl --user stop podman.socket
$ systemctl --user stop podman.service
$ systemctl --user start podman.socket

Note, podman does not support multiple sockets. In other words, there can only be one ListenStream= entry:

link to the relevant source code:

podman/cmd/podman/system/service_abi.go

Lines 43 to 45 in e1496c9

    
           if len(listeners) != 1 { 
        
           	return fmt.Errorf("wrong number of file descriptors for socket activation protocol (%d != 1)", len(listeners)) 
        
           }

diasdmhub · 2024-09-29T17:44:12Z

diasdmhub
Sep 29, 2024
Author

I think I've finally figured out the problem.

It seems that the user's directory permission (/run/user/$UID) is set to 600 like all other users.

[user@pod ~]$ ls -lad /run/user/1004
drwx------ 8 user user 200 Sep 28 18:38 /run/user/1004

This happens because the directory is created with restrictive permissions after user linger is enabled. That is, it is created with user permissions only, no group permissions. Like the example below.

[root@host ~]# sudo loginctl enable-linger 1005
[root@host ~]# su -l test
[test@host ~]$ ls -lad /run/user/1005
drwx------ 3 test test 100 Sep 29 14:20 /run/user/1005

Only after adding execution permission to the directory, other users from the same group were able to access the Podman socket, including Zabbix.

[user@pod ~]$ chmod g+x /run/user/1004/

[user@pod ~]$ ls -lad /run/user/1004/
drwx--x--- 8 user user 200 Sep 28 18:38 /run/user/1004/

[user2@pod ~]$ curl --unix-socket /run/user/1004/podman/podman.sock http://d/v5.0.0/libpod/info
{"host":{"arch":"amd64","buildahVersion":"1.33.8","cgroupManager":"systemd","cgroupVersion":"v2",
(...)
"OsArch":"linux/amd64","Os":"linux"}}

3 replies

piotr-dobrogost Oct 1, 2024

This is exactly the problem I'm asking about in my question at #24074 with the difference that in my case I'm asking about privileged Podman not rootless and I need the access to socket so that my IDE (PyCharm) could use it.
Can somebody please tell me why there is no information on this whereas for Docker it's part of the docs at https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user ?

diasdmhub Oct 1, 2024
Author

I think I see what you mean.
You want to access your root containers with a user other than root. Right?

I think that Podman just doesn't have a specific podman group at the moment, at least not that I could find.

rhatdan Oct 2, 2024
Maintainer

No Podman does not use the podman group, it does not exist.

diasdmhub · 2024-10-02T23:47:36Z

diasdmhub
Oct 2, 2024
Author

Despite adding permissions to the user's run directory and solving the initial permission problem, the permission is reset after a reboot.

So I tried adding the SocketGroup=zabbix line to the user's podman.socket.

[user@pod ~]$ systemctl --user cat podman.socket
# /usr/lib/systemd/user/podman.socket
[Unit]
Description=Podman API Socket
Documentation=man:podman-system-service(1)

[Socket]
ListenStream=%t/podman/podman.sock
SocketMode=0660
SocketGroup=zabbix

[Install]
WantedBy=sockets.target

Reloaded SystemD and restarted the socket, but it did not start. SystemD reported a chown error.

[user@pod ~]$ systemctl --user status podman.socket
× podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-10-02 20:43:33 -03; 1min 5s ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1004/podman/podman.sock (Stream)
        CPU: 790us

Oct 02 20:43:33 pod systemd[113]: Starting Podman API Socket...
Oct 02 20:43:33 pod systemd[1939]: podman.socket: Failed to chown(): Operation not permitted
Oct 02 20:43:33 pod systemd[113]: podman.socket: Control process exited, code=exited, status=235/CHOWN
Oct 02 20:43:33 pod systemd[113]: podman.socket: Failed with result 'exit-code'.
Oct 02 20:43:33 pod systemd[113]: Failed to listen on Podman API Socket.

I must say... using Podman's API socket is turning out to be quite a pain.

3 replies

afbjorklund Oct 3, 2024

If you want to make your changes permanent (like after a reboot), you need to modify the systemd-tmpfiles unit too.

rhatdan Oct 3, 2024
Maintainer

You as a rootless user are attempting to chown a file to a different UID and the kernel is stopping you as expected. The issues you are having is with rootless mode. If you want to do this in rootful mode then it should be fine. You might be able to add the user to the zabbix group but I still don't think the kernel would allow it.

You might need to open a question with systemd on how you could make this happen.

afbjorklund Oct 3, 2024

Seems like it would be easier to run the podman client as the other user (e.g. with sudo), instead of using the socket.

diasdmhub · 2024-10-03T21:15:01Z

diasdmhub
Oct 3, 2024
Author

What I'm trying to accomplish is to monitor the rootless Pod with Zabbix.

So far, what I'm catching up is that SystemD creates the /run/user directory as root.

[root@pod ~]# /usr/lib/tmpfiles.d/systemd.conf
d /run/user 0755 root root -

In it, if user linger is enabled, a directory is created for the user with permission 0700, which means no group access.

[user@pod ~]$ ls -lad /run/user/1004
drwx------ 8 user user 200 Oct  3 16:17 /run/user/1004

Since the /run/user/$UID directory is created by enabling user linger, I've created the .config/user-tmpfiles.d/podman_user_1004.conf file with the following value. (Thanks @afbjorklund)

z /run/user/%U 0710 user user -

With this, I aim to give the directory group execution permission after a reboot.

If we run systemd-tmpfiles manually, it will finally give directory execution permission to.

[user@pod ~]$ SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles --user --create
Looking for configuration files in (higher priority first):
	/etc/xdg/user-tmpfiles.d
	/home/user/.config/user-tmpfiles.d
	/run/user/1004/user-tmpfiles.d
	/home/user/.local/share/user-tmpfiles.d
	/usr/local/share/user-tmpfiles.d
	/usr/share/user-tmpfiles.d
SELinux enabled state cached to: disabled
Reading config file "/usr/share/user-tmpfiles.d/podman-docker.conf"…
Reading config file "/home/user/.config/user-tmpfiles.d/podman_user_1004.conf"…
Running create action for entry z /run/user/1004
Running create action for entry L /run/user/1004/docker.sock
Found existing symlink "/run/user/1004/docker.sock".

Now, users that belong to the rootless Pod user should have access to the socket API.

[user2@pod ~]$ curl --unix-socket /run/user/1004/podman/podman.sock http://d/v5.0.0/libpod/info
{"host":{"arch":"amd64","buildahVersion":"1.33.8","cgroupManager":"systemd","cgroupVersion":"v2","cgroupControllers":
(...)
2024","Built":1725368544,"OsArch":"linux/amd64","Os":"linux"}}

Steps

Just to summarize, these are the steps taken to allow another user access to the rootless Podman socket API.

The rootless Pod user must have linger enabled.

[root@pod ~]# loginctl enable-linger [user|UID]

Append the other user (zabbix in my example) to the rootless user group.

[root@pod ~]# usermod -aG user zabbix
[root@pod ~]# exec newgrp user

Enable the podman.socket for the rootless user.

[user@pod ~]$ systemctl --user enable --now podman.socket

Create the file .config/user-tmpfiles.d/podman_user.conf with the following directive.

z /run/user/%U 0710 user user -

Run systemd-tmpfiles to change permissions on the /run/user/$UID directory.

systemd-tmpfiles --user --create

I've been searching so much the last few days that I may have forgotten a thing or two about these steps.
For now, this is working out for me.

Anyway, this has been quite a learning experience.
Thank you all for your insights.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monitor Podman using unix socket #24103

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 12 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Monitor Podman using unix socket #24103

diasdmhub Sep 29, 2024

Replies: 5 comments · 12 replies

rhatdan Sep 29, 2024 Maintainer

rhatdan Sep 29, 2024 Maintainer

diasdmhub Sep 29, 2024 Author

rhatdan Sep 29, 2024 Maintainer

diasdmhub Sep 29, 2024 Author

diasdmhub Sep 29, 2024 Author

rhatdan Sep 29, 2024 Maintainer

eriksjolund Sep 29, 2024

diasdmhub Sep 29, 2024 Author

piotr-dobrogost Oct 1, 2024

diasdmhub Oct 1, 2024 Author

rhatdan Oct 2, 2024 Maintainer

diasdmhub Oct 2, 2024 Author

afbjorklund Oct 3, 2024

rhatdan Oct 3, 2024 Maintainer

afbjorklund Oct 3, 2024

diasdmhub Oct 3, 2024 Author

Steps

diasdmhub
Sep 29, 2024

Replies: 5 comments 12 replies

rhatdan
Sep 29, 2024
Maintainer

rhatdan Sep 29, 2024
Maintainer

diasdmhub Sep 29, 2024
Author

rhatdan Sep 29, 2024
Maintainer

diasdmhub Sep 29, 2024
Author

diasdmhub
Sep 29, 2024
Author

rhatdan Sep 29, 2024
Maintainer

diasdmhub
Sep 29, 2024
Author

diasdmhub Oct 1, 2024
Author

rhatdan Oct 2, 2024
Maintainer

diasdmhub
Oct 2, 2024
Author

rhatdan Oct 3, 2024
Maintainer

diasdmhub
Oct 3, 2024
Author