Has anybody EVER gotten rootless Podman to work with volumes backed by network attached storage?

[khift363]

I've been researching this problem for 2 weeks now and I'm at my wits end. I've tried everything I can possibly imagine, exhausted every possibility I've come across, and the only conclusion I can come to is that there is no way to make rootless Podman containers run properly with volumes backed by any form of network attached storage. I want to be wrong... but I no longer think I am.

I know what you're going to say -- use UserNS=keep-id and manually tweak things. Yes, if the container is simple enough, or well made and well documented, then yes you can skate by with this. But if the container is complicated, or if it's tossing out chowns left and right during regular runtime and not just setup, then this approach fails, The container I've been doing my proof of concept testing against is Pi-hole, which during runtime will create files as root in a directory owned by root then chown them to 33 -- you can manually tweak permissions so that root can operate in the container fine, and can even create the file, but the moment it tries to chown it the NFS server rejects it because of the UID mismatch and then it's over.

From what I understand, Podman traditionally solves this problem by using SE Linux attributes. So I figured, what if I could get SE Linux attributes working on my NFS share? Kernel v5.9 added xattr support for NFS shares, so according to my research if both client and server are on v5.9 or later and NFS v4.2 is used, xattr support should just be there! Well, turns out, that's true -- my NFS share already supports xattr on files. It's specifically SE Linux that does not work with it, generic xattrs work just fine.

So if NFS doesn't support these things, what about SMB? SMB advertises full SE Linux support. And it does seem to work well -- but there's another problem, because there's always another problem. SMB doesn't use standard Linux filesystem permissions, instead the entire share is owned by one user you set upon mounting the share. Doing a chown on a file in an SMB share will not change the user the file is owned by, but the chown will succeed. SE Linux ensures the container can access and run the file, but... if the container does reasonable self checks on files before accessing them, it'll chown the file to another user, check to see if the chown took, and then error because it doesn't look like it did even though the user owning the file doesn't actually matter. So SMB is out.

Next I tried mounting the NFS share directly into the container volume. I don't even know if this would solve the problem, I get the feeling it's just pushing the tire down the road, but I'd try anything at this point. The kernel recently added support for rootless NFS mounts, and you can do NFS mounts in a named volume, so let's try it why not? Unfortunately, I wasn't able to mount inside the container -- to do a rootless NFS mount you have to set up a record for the mount in /etc/fstab with option 'auto' set, and in order to do that you need to know where the mount is going to mount to. In a container environment that's basically impossible to know ahead of time, so that is right out.

I don't think this is an unreasonable or edge use case. I want to run complicated containers like Nextcloud that store bulk amounts of data. I want to run them rootlessly for security, and I want their data to be stored in my NAS and not on my local drive. I'm willing to change literally anything else in my setup to try and get this to work -- operating system, NAS share, assisting software, etc etc. But it needs to be a solution that works out of the box for each container, now and moving forward. Manually tweaking files to match the container's expected UIDs isn't a viable solution. Not only is it extremely tedious and fraught for mistakes, but the UIDs used by a container is not some stable public API that they can't change without informing their consumers. These UIDs can and will change with updates, and every time it does it's back to the mines manually tweaking Podman containers again.

Please tell me I'm wrong. I really, really want to be wrong. But I just don't see any way forward with this without going rootful. And again, I don't think this is some niche usecase -- I think, honestly, that this usecase is possibly the most common usecase for container usage, because separating storage of state from the actual software is one of the biggest things that containers bring to the picture.

[rhatdan]

[rhatdan]

Ok I read more, chowning on an NFS server is not allowed. NFS does not support User Namespace, so from the NFS servers point of view, it sees USER=$UID trying to chown to a different User.

Nothing Podman can do. Even in rootful Podman this would be blocked, unless the NFS mount point is mounted with root enabled. NFS servers do not trust the clients.

[cshamis]

Pretty sure you're not wrong. I'm trying the same thing. But I read an article from Dan Walsh at RedHat that said it can be done... but what he suggests doesn't actually work. So... I'm hoping I was missing something. Sounds like I'm not.

[rhatdan]

He usually knows what he is talking about. :^)

You can share a volume from network storage, but if you need to chown a file in that directory it will be blocked.
There has been some effort in fuse-overlay to store the content as a single UID and then use different UIDs stored in XATTRs for the file system objects. Not sure if this works with NFS. We have support for this on virtiofs on MACs now for content from it's homedir shared into the container.

[rhatdan]

@giuseppe Thoughts?

[giuseppe]

it won't work out of the box. Volumes work well only with file systems that are user namespace aware. fuse-overlayfs could be used for this use case, but we have not that plugged in for volumes and it requires some tweaking since the upper layer should be handled differently (so not a direct replacement for a standard volume/bind mount)