Running podman inside a docker container with gpu support

[ersguteralbaner]

Issue Description

Hello,
Iam trying to run podman inside a docker container with gpu support. But when I do so I get

Failed to initialize NVML: N/A

Steps to reproduce the issue

Steps to reproduce the issue
1.Run podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L output is GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-98bad0a9-7e4f-c1ef-59e3-8ae46447a327)
2. Run docker exec podmantest podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L output is Failed to initialize NVML: N/A

Describe the results you received

Failed to initialize NVML: N/A

Describe the results you expected

I expect to see Nvidia GPU support. GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-98bad0a9-7e4f-c1ef-59e3-8ae46447a327)

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon_2:2.1.8-0ubuntu22.04+obs16.39_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 98.16
    systemPercent: 0.21
    userPercent: 1.64
  cpus: 16
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2040
  hostname: DESKTOP-C87IB1M
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.133.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 1478873088
  memTotal: 16316170240
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.12-0ubuntu22.04+obs65.12_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.2-0ubuntu22.04+obs12.26_amd64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4243279872
  swapTotal: 4294967296
  uptime: 40h 40m 2.00s (Approximately 1.67 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/p/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 1
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/p/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 21659693056
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/p/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

Yes

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

how was the outer container created?

[ersguteralbaner]

how was the outer container created?

I build a Dockerfile and created an image. Its nothing specific iam trying to get podman running with gpu support in a Docker container.
Thats the Dockerfile

FROM quay.io/podman/stable:v4.5.0

# nvidia toolkit
RUN curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo \
 && yum -y install nvidia-container-toolkit \
 && yum -y clean all \
 && rm -fr /var/cache

# docker alias
RUN ln -s /usr/bin/podman /usr/local/bin/docker

# entrypoint
COPY ./entrypoint.sh /
RUN chmod +x entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

[giuseppe]

docker exec podmantest implies there is already a container podmantest created. How was it created?