sudo in non-native container complains "effective uid is not 0" #20445

rgov · 2023-10-23T00:33:09Z

rgov
Oct 23, 2023

Issue Description

Running sudo as the non-root user in a container for the native architecture works fine. However when running a non-native container, the sudo command fails.

Steps to reproduce the issue

Examples via this StackOverflow question from @bwoodsend.

In this example, my native architecture is Arm64.

$ cat Dockerfile 
FROM alpine

RUN apk add shadow sudo
RUN echo '%wheel ALL=(ALL:ALL) NOPASSWD: ALL' >> /etc/sudoers
RUN useradd --create-home --non-unique --uid 1000 --groups wheel user

$ podman run --platform=linux/arm64 -it --user=1000:wheel \
    $(podman build -q --platform=linux/arm64 .) \
    sudo whoami
root

$ podman run --platform=linux/amd64 -it --user=1000:wheel \
    $(podman build -q --platform=linux/amd64 .) \
    sudo whoami
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

Describe the results you received

In the native container, the user with ID 1000 is able to invoke sudo whoami successfully. In the non-native container, sudo complains.

Describe the results you expected

sudo should work for non-native containers.

podman info output

host:
  arch: arm64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 88.57
    systemPercent: 6.14
    userPercent: 5.29
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2034
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.5-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1540485120
  memTotal: 1980387328
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.aarch64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 17.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 5402329088
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839065
  BuiltTime: Wed Sep 27 11:24:25 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/arm64
  Version: 4.7.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

A number of workarounds were tried in the above-linked StackOverflow post, including running with a rootful machine VM; none worked.

Luap99 · 2023-10-23T09:06:16Z

Luap99
Oct 23, 2023
Maintainer

also see #18425
Emulation is provided via qemu-user-static so moving this to a discussion as it is not a podman bug.

4 replies

bwoodsend Oct 23, 2023

It works with Docker (although you have to remember to add a --credential yes to the magic docker run --rm --privileged multiarch/qemu-user-static command). Is that not an indicator that the issue is specific to podman?

rhatdan Oct 24, 2023
Maintainer

I think it is an indicator that the machine that you are running on has different configurations of the kernel or different kernels.

Basically this looks like the the OS attempting to do the setuid operation is denied via the kernel. If there is some kernel setting to set to allow this behaviour, then perhaps podman machine should configure it.

nalind Oct 24, 2023
Maintainer

Keep in mind that per the docs, the "C" flag, which "--credential yes" sets along with the "O" flag, tells the kernel to run the emulator itself with elevated privileges when the binary that the emulator is being invoked to run has the setuid/setgid bit set.

rgov Oct 24, 2023
Author

The implication of which is that a vulnerability in QEMU, when invoking a setuid binary with malicious input, could potentially gain root privileges. But this seems like a relatively minor risk:

A user that can invoke containers can already run a container with native setuid binaries, so why bother exploiting the interpreter?
1. Therefore the only attack scenario is that input comes from an external user who cannot spawn containers, e.g., a web app that passes user input to a non-native setuid program in a container, and that input is capable of exploiting a vulnerability in the interpreter rather than the emulated program itself, or something about the emulation makes the program easier to exploit. While plausible this seems far-fetched.
The proposed configuration is the same as Docker Desktop, so it's not lowering security relative to comparable software.
This is only for the podman machine VM which is isolated from the (usually Mac or Windows) host.

rgov · 2023-10-23T15:19:45Z

rgov
Oct 23, 2023
Author

Does an incompatibility with Docker Desktop count as a podman bug? I believe the VM used by Docker Desktop on Mac configures binfmt_misc in a way that allows this to work.

Is the podman machine image customized for podman? If so ought we be able to change the config file?

6 replies

rgov Oct 24, 2023
Author

Ok, let's add the C flag. From the documentation:

C - credentials
Currently, the behavior of binfmt_misc is to calculate the credentials and security token of the new process according to the interpreter. When this flag is included, these attributes are calculated according to the binary. It also implies the O flag. This feature should be used with care as the interpreter will run with root permissions when a setuid binary owned by root is run with binfmt_misc.

To enable the flag, I copied each file from /usr/lib/binfmt.d to /etc/binfmt.d, which is the documented way of overriding the configuration.

$ podman machine set --rootful=false
$ podman machine start
$ podman machine ssh

(machine)$ for x in /usr/lib/binfmt.d/*.conf; do \
    sed 's/\(:[^C:]*\)$/\1C/' "$x" | \
    sudo tee /etc/binfmt.d/"$(basename "$x")"; \
done

$ podman machine stop
$ podman machine start

# Expected result
$ podman run --platform=linux/amd64 -it --user=1000:wheel \
    $(podman build -q --platform=linux/amd64 .) \
    sudo whoami
root

It looks like this resolves the issue. Is it possible to get this made to the default system image?

Luap99 Oct 25, 2023
Maintainer

@baude @ashley-cui WDYT? Is this something podman machine could setup with ignition?

rhatdan Oct 25, 2023
Maintainer

If this is a security issue, we might want to make it opt in.

ashley-cui Oct 25, 2023
Maintainer

@baude @ashley-cui WDYT? Is this something podman machine could setup with ignition?

I believe so!

BhawaniSingh Jul 22, 2024

Ok, let's add the C flag. From the documentation:

C - credentials
Currently, the behavior of binfmt_misc is to calculate the credentials and security token of the new process according to the interpreter. When this flag is included, these attributes are calculated according to the binary. It also implies the O flag. This feature should be used with care as the interpreter will run with root permissions when a setuid binary owned by root is run with binfmt_misc.

To enable the flag, I copied each file from /usr/lib/binfmt.d to /etc/binfmt.d, which is the documented way of overriding the configuration.
$ podman machine set --rootful=false
$ podman machine start
$ podman machine ssh

(machine)$ for x in /usr/lib/binfmt.d/*.conf; do \
    sed 's/$:[^C:]*$$/\1C/' "$x" | \
    sudo tee /etc/binfmt.d/"$(basename "$x")"; \
done

$ podman machine stop
$ podman machine start

# Expected result
$ podman run --platform=linux/amd64 -it --user=1000:wheel \
    $(podman build -q --platform=linux/amd64 .) \
    sudo whoami
root
It looks like this resolves the issue. Is it possible to get this made to the default system image?

I'm on mac m3 and this fix help me fixing the sudo issue, it would be great if we can have this in podman machine by default

bwoodsend · 2023-10-23T20:06:21Z

bwoodsend
Oct 23, 2023

I don't actually see this issue anymore. Sudo+podman+qemu are working fine together on both my local Arch Linux machine and Ubuntu+GitHub Actions (using the PPA to get something semi recent).

podman info

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 97.99
    systemPercent: 0.54
    userPercent: 1.47
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: manjaro-2212
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.5.5-1-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 217669632
  memTotal: 8046858240
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.8.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.9.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 1789652992
  swapTotal: 4294963200
  uptime: 50h 57m 43.00s (Approximately 2.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/brenainn/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/brenainn/.local/share/containers/storage
  graphRootAllocated: 126227120128
  graphRootUsed: 118104498176
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 42
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/brenainn/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.1
  Built: 1696583554
  BuiltTime: Fri Oct  6 10:12:34 2023
  GitCommit: ef83eeb9c7482826672f3efa12db3d61c88df6c4-dirty
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.1

0 replies

rhatdan · 2023-10-24T23:41:29Z

rhatdan
Oct 24, 2023
Maintainer

Are you saying that docker run --credential yes ... is a valid command? Perhaps hidden?

5 replies

bwoodsend Oct 24, 2023

No, docker run --rm --privileged multiarch/qemu-user-static --credential yes is the command I meant. i.e. the flag is interpreted by the multiarch/qemu-user-static entry point rather than docker itself.

rgov Oct 25, 2023
Author

By running multiarch/qemu-user-static with --privileged, I think you're allowing the container to modify the host kernel's binfmt_misc config.

The entry point you mentioned runs this script which updates the host's binfmt_misc registrations for QEMU. The script handles the --credential yes argument to enable the flags OC on the binfmt_misc registration. Note that C implies O already.

So basically, when you run multiarch/qemu-user-static you're just allowing it to reconfigure your host for you, rather than updating the host's binfmt_misc config directly (e.g., through the systemd-binfmt conf files). The reconfiguration it does is basically the same as I describe here.

rhatdan Oct 25, 2023
Maintainer

Doesn't that same command work for podman run on a Mac or Windows?

rhatdan Oct 25, 2023
Maintainer

It probably requires you run in rootful mode.

rgov Oct 25, 2023
Author

Sure, in theory you could. (On my system the VM hangs after multiarch/qemu-user-static does its magic, but I am reasonably confident that could be overcome.)

However, if you need a workaround, I don't see any reason to prefer this workaround over podman machine ssh and patching the VM config directly. It seems not great to direct users to run a third-party container in privileged mode to semi-magically mess with the podman VM kernel. Another weird aspect of multiarch/qemu-user-static is that it persistently (across multiple container lifespans) keeps that container's QEMU binaries loaded after the container is stopped—overriding the podman VM's—and effects subsequently launched containers. It's hard to imagine this being the supported solution.

Anyway, the point is that this is an incompatibility with Docker Desktop; containers that work on Docker Desktop do not work on out-of-the-box podman. The Docker Desktop VM registers formats with the POCF flags whereas podman just uses F.

Users of Raspberry Pis and Macs are most likely to encounter this compatibility issue because these platforms more heavily rely on non-native platform containers.

# Docker Desktop on Mac
# cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /usr/bin/qemu-x86_64
flags: POCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

The P flag also has a minor effect that might conceivably cause an incompatibility.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sudo in non-native container complains "effective uid is not 0" #20445

{{title}}

Replies: 4 comments 15 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

sudo in non-native container complains "effective uid is not 0" #20445

rgov Oct 23, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Replies: 4 comments · 15 replies

Luap99 Oct 23, 2023 Maintainer

bwoodsend Oct 23, 2023

rhatdan Oct 24, 2023 Maintainer

nalind Oct 24, 2023 Maintainer

rgov Oct 24, 2023 Author

rgov Oct 23, 2023 Author

rgov Oct 24, 2023 Author

Luap99 Oct 25, 2023 Maintainer

rhatdan Oct 25, 2023 Maintainer

ashley-cui Oct 25, 2023 Maintainer

BhawaniSingh Jul 22, 2024

bwoodsend Oct 23, 2023

rhatdan Oct 24, 2023 Maintainer

bwoodsend Oct 24, 2023

rgov Oct 25, 2023 Author

rhatdan Oct 25, 2023 Maintainer

rhatdan Oct 25, 2023 Maintainer

rgov Oct 25, 2023 Author

rgov
Oct 23, 2023

Replies: 4 comments 15 replies

Luap99
Oct 23, 2023
Maintainer

rhatdan Oct 24, 2023
Maintainer

nalind Oct 24, 2023
Maintainer

rgov Oct 24, 2023
Author

rgov
Oct 23, 2023
Author

rgov Oct 24, 2023
Author

Luap99 Oct 25, 2023
Maintainer

rhatdan Oct 25, 2023
Maintainer

ashley-cui Oct 25, 2023
Maintainer

bwoodsend
Oct 23, 2023

rhatdan
Oct 24, 2023
Maintainer

rgov Oct 25, 2023
Author

rhatdan Oct 25, 2023
Maintainer

rhatdan Oct 25, 2023
Maintainer

rgov Oct 25, 2023
Author