Rootless podman, NFS volumes and the -w flag not working

[woopla]

Hello,

I have an ongoing project to provide build capacity for multiple RHEL versions from a single RHEL 8 host (this is for a binary shipped to customers and supported on RHEL6,7,8). This used to be working with Docker, but since Podman is better supported on RHEL 8, I want to switch to Podman.

I can almost run the script unchanged with podman-docker installed, except for the -w option:

> docker  run --rm -it -v /<NFS path A>:/<NFS path A>:rshared,z --net=host --user $UID:$GID --userns=keep-id -w $PWD <registry>/centos6-dts6:latest bash
Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: mkdir <NFS location within NFS path A >: permission denied: OCI permission denied

Without using -w I can get in the container just fine, and the NFS location ownership looks correct - I can write to it and it appears as the correct owner outside of the container (NFS shares are mounted with root_squash).

I'm running what's currently available on RHEL 8.4:

> podman version
Version:      3.0.2-dev
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Wed Apr  7 01:36:54 2021
OS/Arch:      linux/amd64

Not sure if this is a bug or limitation in Podman rootless, or if I need to change something in the options there...

Thanks,

Clément

[woopla]

By the way, I also need the --group-add keep-groups option, which is only available with Podman 3.2. When can we expect this version to be available in RHEL 8.4?

[vrothberg]

I am bad with dates but I am sure that @fatherlinux and @TomSweeneyRedHat will know.

[rhatdan]

Should be in RHEL8.4.0.2, which I believe was recently released.

[woopla]

Thanks guys, I'll check with my IT to see where we're at with our internal repos.

[woopla]

I was able to update podman to 3.2.3 which made the option available, but no luck on my initial issue with -w.

[vrothberg]

Thanks for reaching out, @woopla!

container init caused: mkdir ...: permission denied: OCI permission denied

Does the path you specify in -w exist on the image? The mkdir indicates that it does not.

[woopla]

Hey @vrothberg the path does not exist in the image, it is one of the NFS paths mounted through -v. Looks like it's checking for the existence of the path before mounting the volume(s), then?

[rhatdan]

With this command it looks like User WOOPLA is attempting to create a directory owned by $UID within the user namespace on an NFS share and it is being blocked, because WOOPLA is not allowed to change ownership of a directory. NFS Server does not understand User Namespace would be my assumption.

[woopla]

Hey @rhatdan the owner of that path is the same one that is running inside the container. I checked this by running without -w, changing to the location and running mkdir in the same parent directory. That worked fine.

[woopla]

I was able to work on this some more today: the problem is one of permissions indeed, but that's because --group-add keep-groups does not do what I expect it to :)

> id -G
6236 205 235 6043 6154 8551 8600 50043 51171 51434 51438
> podman --runtime /usr/bin/crun run --rm -it --user="$(id -ur):$(id -gr)" --userns=keep-id --group-add=keep-groups debian:stable-slim id -G
6236 65534

I also tried to pass --add-group manually for all groups (what I've been doing with Docker CE thus far), and it fails to start altogether:

> podman --runtime /usr/bin/crun run --rm -it --user="$(id -ur):$(id -gr)" --userns=keep-id --group-add 205 --group-add 6043 --group-add 6154 --group-add 100224 --group-add 235 --group-add 51438 --group-add 50043 --group-add 51434 --group-add 51171 --group-add 8551 --group-add 8600  debian:stable-slim id -G
Error: OCI runtime error: setgroups: Invalid argument

(same thing happens if I do --group-add="205,6043,6154,100224,235,51438,50043,51434,51171,8551,8600").

@rhatdan and @vrothberg what should I change to the arguments for this to work? I need the groups inside the container to match (or be a superset of) the ones on the host.

[woopla]

OK then is there a solution to map all groups inside the container, like it's done for the primary group? Docker can do it, but I suppose it's easier for them since it's running the process as root, correct (even for their version of rootless with users being members of the docker Unix group)?

[rhatdan]

You could do this, but that is not required, the access to the groups still exists.

Doing this in rootless mode is mode difficult, because you need to update the /etc/subgid with all of the groups, This is not something I would recommend.

[woopla]

Yeah but there's something I'm missing here:

I think I should be able to get by with just two groups (my primary group, and the other one the data is under). Using --user="$(id -ur):235" --userns=keep-id --group-add=keep-groups works (235 is the extra group ID I want). This only works for my case though, I have 450 other people this has to work with :)

From the documentation, I should be able to map 65536 groups from the single /etc/subgid entry of clement:506236:65536. I tried using the lowest group ID of the bunch as the --user GID, no dice... Could you please give me an example of mapping three groups?

[rhatdan]

@giuseppe Could you help with this. I am on PTO this week.

[giuseppe]

the values you pass to --group are relative to the user namespace.

The mapping done by Podman is $UID -> 0, each other entry in the /etc/sub?id file is mapped starting at 1.

So if you want to use group 506236 on the host, you'll need to specify --group-add 1

Once you add a group though, you lose access to all the external unmapped groups injected with --group-add=keep-groups.

Once a setgroups() is called from the container, all the current groups are lost. Since the external unmapped groups are not present in the user namespace, there is no way to use them with setgroups().

[rhatdan]

Run sleep inside of the container and then outside of the container look at the UID and GIDs of the process running sleep.
grep -i uid /proc/PID/status
grep -i Groups /proc/PID/status

That should give you information on the process attempting to do the mkdir.