Any idea about this problem: "Error: error creating container storage: could not find enough available IDs" ?

[patrickdung]

podman V3.4.4 on FC35, rootless.
Problem occurs when '--userns=auto' is added
podman run --name=container-gadget-root -d --label io.podman.compose.config-hash=123 --label io.podman.compose.project=pod-gadgets-root --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=pod-gadgets-root --label com.docker.compose.project.working_dir=/data/infra/home/podman/pod-gadgets-root --label com.docker.compose.project.config_files=docker-compose.yaml --label com.docker.compose.container-number=1 --label com.docker.compose.service=service-gadget-root --cap-drop all --net pod-gadgets-root_default --network-alias service-gadget-root --hostname container-gadget-root user1/pod-gadgets-root:0.2 --userns=auto
Error: error creating container storage: could not find enough available IDs

# cat /etc/subgid
user1:100000:65536
root:165536:65536

# cat /etc/subuid
user1:100000:65536
root:165536:65536

Thanks in advance.

[mheon]

I don't think --userns=auto is supported with rootless - @giuseppe can you confirm?

[giuseppe]

it works. It is limited to the IDs allocated to the rootless user though

[rhatdan]

It will work in rootless, BUT you must have enough UIDs to subdevide them. If the mapping as a UID not mapped into the user namespace it will fail.

If you try --userns=auto:size=5000 it is more likely to work.

[patrickdung]

Ok, '--userns=auto' should be put right after 'podman run'.
But I tried with '--userns=auto:size=5000', '--userns=auto' and other values, the error is the same.

[giuseppe]

you probably have other containers that are running without any user namespace, so they are already using all the available IDs. --userns=auto fails because it cannot find any unused IDs for the new container.

[patrickdung]

@giuseppe
Would you provide commands on how to check what containers are running without any user namespaces? Is it possible to check or increase what is the (ID/namespace) limit?

[giuseppe]

just run podman ps -a and inspect each of them.

You can try to add more IDs to your user (through /etc/subid and /etc/subgid) then run podman system migrate

[patrickdung]

Here's my testing result:

1. I wipe out all the containers and pods.
   (podman ps -a and podman pod ps -a , shows nothing)

2. After that I increase the ID in /etc/subuid and subgid.

3. And then run 'podman system migrate'.

4. Two testings:

• Test 1: First run a pod with '--userns=auto', it's succesful.

• Test 2: First run a pod with 'PODMAN_USERNS=keep-id' ( I need this for the code-server to access my local user files)
  Then start another pod with 'PODMAN_USERNS=auto' / 'PODMAN_USERNS=auto:size=10'
  The error message "Error: error creating container storage: could not find enough available IDs" appears again.

So, is it by design that once 'keep-id' is used, it can't use the 'auto' option?

[rhatdan]

keep-id must be using the entire range of available UIDs. So auto would no longer be available.

@giuseppe do we have the same with rootful? Should we never use more then 65k when running in a user namespace? Should this be configurable in containers.conf?

[giuseppe]

With root we don't have this problem I think, since keep-id is not really useful to use.

[rhatdan]

Right, but the issue is the size of the range of containers that --userns=keep-id takes, If it only took a set amount (64k) then other user namespaces might be able to run. Or maybe we should add parsing to --userns=keep-id:size=5000 or something like that.