rootless_storage_path results in OCI permission denied #13230

will · 2022-01-11T17:30:10Z

will
Jan 11, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

I think? I've done a lot of searching, and don't see much info on people using rootless_storage_path so it could just be me using it wrong. At the very least hopefully this puts more things to search out there on this setting.

Description

I'm trying to move rootless container storage to a different drive, but after making that change, I can no longer start containers.

Steps to reproduce the issue:

I left all the uncommented default settings for storage.conf but turned on rootless_storage_path

# /etc/containers/storage.conf
[storage]
# (default) Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# (default) Temporary storage location
runroot = "/run/containers/storage"

# (default) Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# (custom) Storage path for rootless users
rootless_storage_path = "/dat/containers/storage/$USER"

[storage.options.overlay]
# (default) mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

Then I create a directory for a user, in this example the user happens to be named "podman", but I dont think it matters.

sudo install -d -o podman -g podman -m 750 /dat/containers/storage/podman

Then try to start a container

podman run -dt -p 6379:6379/tcp docker.io/redis

Describe the results you received:

Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/dat/containers/storage/podman/volumes/147d937067e8977b5b1bf9576f18a5845afd652d33e91df94197c994fd8478e3/_data" to rootfs at "/data" caused: operation not permitted: OCI permission denied

Describe the results you expected:

The container to start. It starts fine when I don't have the config change to move the rootless location.

I have also tried to disable SELinux, but it did not change the situation.

Output of podman version:

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.7
Built:        Tue Nov  9 12:25:58 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module_el8.5.0+2586+018f24d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 605b895b7c7a95097901b9db7a9058f0f362d34c'
  cpus: 2
  distribution:
    distribution: '"almalinux"'
    version: "8.5"
  eventLogger: file
  hostname: kpgvjriyejdqhoigkgprudqbcy
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-348.7.1.el8_5.x86_64
  linkmode: dynamic
  memFree: 4750610432
  memTotal: 8240283648
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module_el8.5.0+2586+018f24d7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module_el8.5.0+2586+018f24d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 1073737728
  swapTotal: 1073737728
  uptime: 24h 17m 44.39s (Approximately 1.00 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-1.module_el8.5.0+2586+018f24d7.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /dat/containers/storage/podman
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /tmp/podman-run-1001/containers
  volumePath: /dat/containers/storage/podman/volumes
version:
  APIVersion: 3.3.1
  Built: 1636460758
  BuiltTime: Tue Nov  9 12:25:58 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.3.1-9.module_el8.5.0+2586+018f24d7.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

I think this is at least very close to the latest version.

For the guide, I feel this is maybe related to https://github.com/containers/podman/blob/main/troubleshooting.md#7-permission-denied-when-running-podman-commands but I'm not sure?

Additional environment details (AWS, VirtualBox, physical, etc.):

AWS

rhatdan · 2022-01-11T20:04:09Z

rhatdan
Jan 11, 2022
Maintainer

First thought would be SELinux?
$ sudo semanage fcontext -a -e $HOME/.local/share/containers /dat/containers
$ restorecon -R -v /dat

0 replies

will · 2022-01-11T20:16:49Z

will
Jan 11, 2022
Author

Thanks @rhatdan I tried to turn off SELinux with setenforce 0. The behavior seems the same with both 1 and 0

[podman@kpgvjriyejdqhoigkgprudqbcy ~]$ getenforce
Enforcing
[podman@kpgvjriyejdqhoigkgprudqbcy ~]$ podman run -dt -p 6379:6379/tcp docker.io/redis
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/dat/containers/storage/podman/volumes/0b75c2e026ad9b53d911f8a96b731e2b55dec0f38071e4aa776d81ba084b192d/_data" to rootfs at "/data" caused: operation not permitted: OCI permission denied

[podman@kpgvjriyejdqhoigkgprudqbcy ~]$ getenforce
Permissive
[podman@kpgvjriyejdqhoigkgprudqbcy ~]$ podman run -dt -p 6379:6379/tcp docker.io/redis
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/dat/containers/storage/podman/volumes/aaf0919b5487e47b91dd7de3702192b42022ee56413825d786c6efe278998f4a/_data" to rootfs at "/data" caused: operation not permitted: OCI permission denied

And just to be sure I tried the two commands you gave, but it also has the same error after trying to do podman run

0 replies

rhatdan · 2022-01-11T20:20:00Z

rhatdan
Jan 11, 2022
Maintainer

Does podman version of podman info work?

0 replies

rhatdan · 2022-01-11T20:22:03Z

rhatdan
Jan 11, 2022
Maintainer

And is /dat directory and its children owned by the podman user?

0 replies

will · 2022-01-12T10:11:49Z

will
Jan 12, 2022
Author

Does podman version of podman info work?

I'm not sure what this mean, unless "of" a typo of "or". If that is the case, yes both work. I have the output of both in the first post. The output for podman info --debug is in a details details section since it's long. If you mean some other command though, please let me know.

And is /dat directory and its children owned by the podman user?

No, everything is root up until the that directory. However I think the permissions of each directory is correct at every level:

drwxr-xr-x. 7  root     root     4096 Jan 10 14:51 dat
drwxr-xr-x. 3  root     root     4096 Jan 10 14:51 containers
drwxr-xr-x. 5  root     root     4096 Jan 11 12:23 storage
drwxr-x---. 10 podman   podman   4096 Jan 11 15:37 podman

and then everything in the final directory seem okay to me. They were all created by trying to run the first redis container:

$ ls -la /dat/containers/storage/podman/
total 44
drwxr-x---. 10 podman podman 4096 Jan 11 15:37 .
drwxr-xr-x.  5 root   root   4096 Jan 11 12:23 ..
drwx------.  2 podman podman 4096 Jan 11 12:23 libpod
drwx------.  2 podman podman 4096 Jan 11 12:23 mounts
drwx------. 15 podman podman 4096 Jan 11 20:16 overlay
drwx------.  8 podman podman 4096 Jan 11 20:16 overlay-containers
drwx------.  3 podman podman 4096 Jan 11 15:37 overlay-images
drwx------.  2 podman podman 4096 Jan 11 20:16 overlay-layers
-rw-r--r--.  1 podman podman   64 Jan 11 20:16 storage.lock
drwx------.  2 podman podman 4096 Jan 11 12:23 tmp
-rw-r--r--.  1 podman podman    0 Jan 11 12:23 userns.lock
drwx------.  8 podman podman 4096 Jan 11 20:16 volumes

0 replies

will · 2022-01-12T10:34:12Z

will
Jan 12, 2022
Author

And then going for each level of the path in the error message /dat/containers/storage/podman/volumes/750a724b32fd9cac637b0e828a2184a54d1bd6aa7e3f202937c7b45cee5256e7/_data

drwx------.  9 podman podman 4096 Jan 12 10:26 volumes
drwx------.  3 podman podman 4096 Jan 12 10:26 750a724b32fd9cac637b0e828a2184a54d1bd6aa7e3f202937c7b45cee5256e7
drwxr-xr-x.  2 166534 166534 4096 Dec 21 12:42 _data

And then _data is empty

The owner of the _data directory is suspicious to me

Here is the subuid and subgid

$ cat /etc/subuid
ec2-user:100000:65536
postgres:231072:65536
podman:165536:65536
$ cat /etc/subgid
ec2-user:100000:65536
postgres:231072:65536
podman:165536:65536

postgres is in there from me trying earlier to run a container as the postgres system user which didn't have a home directory at all. Then I created the podman user as a normal user with a home directory. I did manually add entriesof these files for postgres, but did not manually add entries for podman. I suppose they showed up when I created the podman user.

0 replies

will · 2022-01-12T13:19:20Z

will
Jan 12, 2022
Author

I tried running podman system reset both as the podman user and with sudo, then completely deleted /dat/containers/storage/podman and recreated it with sudo install -d -o podman -g podman -m 750 /dat/containers/storage/podman.

Then running podman run -dt -p 6379:6379/tcp docker.io/redis

Trying to pull docker.io/library/redis:latest...
Getting image source signatures
Copying blob c7a4e4382001 done
Copying blob 4044b9ba67c9 done
Copying blob c8388a79482f done
Copying blob a2abf6c4d29d done
Copying blob 1abfd3011519 done
Copying blob 413c8bb60be2 done
Copying config 7614ae9453 done
Writing manifest to image destination
Storing signatures
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/dat/containers/storage/podman/volumes/b473f3d4e713d659fbd16e97771ebdb19f55eeacc399dd2e39ee0874104c40fa/_data" to rootfs at "/data" caused: operation not permitted: OCI permission denied

So the hash is different, which is reasonable. But the _data dir is still owned by 166534

ls -la /dat/containers/storage/podman/volumes/b473f3d4e713d659fbd16e97771ebdb19f55eeacc399dd2e39ee0874104c40fa/
total 12
drwx------. 3 podman podman 4096 Jan 12 13:16 .
drwx------. 3 podman podman 4096 Jan 12 13:16 ..
drwxr-xr-x. 2 166534 166534 4096 Dec 21 12:42 _data

0 replies

rhatdan · 2022-01-13T13:37:09Z

rhatdan
Jan 13, 2022
Maintainer

podman unshare cat /proc/self/uid_map

I think something is going wrong in your user namespace.

Can you chown the _data directory to be owned by the Podman user.

How are you becoming the podman user?

0 replies

will · 2022-01-13T14:33:31Z

will
Jan 13, 2022
Author

podman unshare cat /proc/self/uid_map

 $ podman unshare cat /proc/self/uid_map
         0       1001          1
         1     165536      65536

Can you chown the _data directory to be owned by the Podman user.

$ ls -la /dat/containers/storage/podman/volumes/b473f3d4e713d659fbd16e97771ebdb19f55eeacc399dd2e39ee0874104c40fa/_data
total 8
drwxr-xr-x. 2 podman podman 4096 Dec 21 12:42 .
drwx------. 3 podman podman 4096 Jan 12 13:16 ..
$ podman run -dt -p 6379:6379/tcp docker.io/redis
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/dat/containers/storage/podman/volumes/dd3654cdd5959f80647afabcbd53adbbdb779ea8ad965f4032da1bcaee85d75c/_data" to rootfs at "/data" caused: operation not permitted: OCI permission denied

How are you becoming the podman user?

Right now sudo su podman.

However my main goal is to eventually have a systemd socket for the daemon owned by this user, so that I can do podman --remote --url unix:///run/podman/podman.sock from any other local user to start rootless containers, but just have them all under a single user. That was more-or-less working until I tried to move the storage location. (Though if the socket is problematic, doing sudo -u podman podman <blah> is okay`, I'm not married to the idea of the systemd socket)

0 replies

2022-02-13T00:04:53Z

github-actions[bot]
bot Feb 13, 2022

A friendly reminder that this issue had no activity for 30 days.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rootless_storage_path results in OCI permission denied #13230

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

rootless_storage_path results in OCI permission denied #13230

will Jan 11, 2022

Replies: 10 comments

rhatdan Jan 11, 2022 Maintainer

will Jan 11, 2022 Author

rhatdan Jan 11, 2022 Maintainer

rhatdan Jan 11, 2022 Maintainer

will Jan 12, 2022 Author

will Jan 12, 2022 Author

will Jan 12, 2022 Author

rhatdan Jan 13, 2022 Maintainer

will Jan 13, 2022 Author

github-actions[bot] bot Feb 13, 2022

will
Jan 11, 2022

rhatdan
Jan 11, 2022
Maintainer

will
Jan 11, 2022
Author

rhatdan
Jan 11, 2022
Maintainer

rhatdan
Jan 11, 2022
Maintainer

will
Jan 12, 2022
Author

will
Jan 12, 2022
Author

will
Jan 12, 2022
Author

rhatdan
Jan 13, 2022
Maintainer

will
Jan 13, 2022
Author

github-actions[bot]
bot Feb 13, 2022