Failed to add conmon to cgroupfs sandbox cgroup #21901

djarbz · 2024-02-29T14:31:34Z

djarbz
Feb 29, 2024

Issue Description

I installed Podman per the wiki and I am unable to get any container to run.
Unfortunately, Cgroups is beyond my scope of knowledge to properly troubleshoot and the help I have received in the OpenWRT repo has not been able to resolve the issue.

Steps to reproduce the issue

Install podman and dependencies per the WIKI linked above.
podman run hello-world

Describe the results you received

WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
Error: OCI runtime error: crun: writing file /sys/fs/cgroup/cgroup.subtree_control: Invalid argument

Describe the results you expected

Container should run without issue.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 99.14
    systemPercent: 0.52
    userPercent: 0.33
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: openwrt
    version: 23.05.0
  eventLogger: none
  freeLocks: 2045
  hostname: OpenWrt
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.134
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1660133376
  memTotal: 1983795200
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.9.0
    package: Unknown
    path: /usr/lib/podman/netavark
    version: netavark 1.9.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: "crun version 1.12\ncommit: \nrundir: /run/crun\nspec: 1.0.0\n+SELINUX
      +APPARMOR +CAP +SECCOMP +EBPF +YAJL"
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 35h 10m 49.00s (Approximately 1.46 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /srv/.podman/images
    overlay.mountopt: nodev
  graphRoot: /srv/.podman/storage
  graphRootAllocated: 59237908480
  graphRootUsed: 920846336
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /srv/.podman/storage/volumes
version:
  APIVersion: 4.8.0
  Built: 1706521279
  BuiltTime: Mon Jan 29 03:41:19 2024
  GitCommit: ""
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

[email protected] x86 on Sophos SG-105r1

Additional information

OpenWRT Issue: openwrt/packages#23544

root@OpenWrt:/srv/.podman/scripts# ls /sys/fs/cgroup/
cgroup.controllers      cgroup.stat             cpuset.cpus.effective   memory.stat
cgroup.max.depth        cgroup.subtree_control  cpuset.mems.effective   services
cgroup.max.descendants  cgroup.threads          io.stat
cgroup.procs            cpu.stat                libpod_parent

root@OpenWrt:~# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory pids rdma

runc seems to at least launch the container, but still throws that error.

root@OpenWrt:~# podman run --rm --runtime=runc hello-world:latest
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@OpenWrt:~# podman run --rm --runtime=crun hello-world:latest
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument

crun with DEBUG logging

root@OpenWrt:~# podman --log-level debug run --rm --runtime=crun hello-world
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm --runtime=crun hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /srv/.podman/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /srv/.podman/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /srv/.podman/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/srv/.podman/images
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend none
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime uxc initialization failed: no valid executable found for OCI runtime uxc: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded network podman: &{podman 5ef894788befd4d42498314b6e66282ca730aa2e1e82f9b9597bf4d1725ca074 bridge podman0 2023-02-20 08:56:34.652030952 +0000 UTC [{{{10.127.127.0 ffffff00}} 10.127.127.254 <nil>}] [] false false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello-world (policy: missing)
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/hello-world:latest" ...
DEBU[0000] reference "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Looking up image "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "docker.io/library/hello-world:latest" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "docker.io/library/hello-world:latest" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "localhost/hello-world:latest" ...
DEBU[0000] reference "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 3 for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9"
DEBU[0000] Container "f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9" has work directory "/srv/.podman/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata"
DEBU[0000] Container "f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9" has run directory "/run/containers/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating!        PID=4928
DEBU[0000] Enabling signal proxying
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/srv/.podman/storage/overlay/l/LZHX62KQRI3NH3KXXGKPFC2BIO,upperdir=/srv/.podman/storage/overlay/6b9cb0043d7d178bbcbab61291b9c887081f551db2a46ccdcdef299ae61ead2b/diff,workdir=/srv/.podman/storage/overlay/6b9cb0043d7d178bbcbab61291b9c887081f551db2a46ccdcdef299ae61ead2b/work,nodev,volatile
DEBU[0000] Made network namespace at /run/netns/netns-e5ea4075-8414-c6f5-b435-8c1944782164 for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9
DEBU[0000] Mounted container "f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9" at "/srv/.podman/storage/overlay/6b9cb0043d7d178bbcbab61291b9c887081f551db2a46ccdcdef299ae61ead2b/merged"
DEBU[0000] Created root filesystem for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 at /srv/.podman/storage/overlay/6b9cb0043d7d178bbcbab61291b9c887081f551db2a46ccdcdef299ae61ead2b/merged
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[DEBUG netavark::firewall] Forcibly using firewall driver none
[INFO  netavark::firewall] Not using firewall
[DEBUG netavark::network::bridge] Setup network podman
[DEBUG netavark::network::bridge] Container interface name: eth0 with IP addresses [10.127.127.18/24]
[DEBUG netavark::network::bridge] Bridge name: podman0 with IP addresses [10.127.127.254/24]
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth0/autoconf to 0
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth0/arp_notify to 1
[INFO  netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 10.127.127.254, metric 100)
[DEBUG netavark::dns::aardvark] Spawning aardvark server
[DEBUG netavark::dns::aardvark] start aardvark-dns: ["/usr/lib/podman/aardvark-dns", "--config", "/run/containers/networks/aardvark-dns", "-p", "53", "run"]
[DEBUG netavark::commands::setup] {
        "podman": StatusBlock {
            dns_search_domains: Some(
                [
                    "dns.podman",
                ],
            ),
            dns_server_ips: Some(
                [
                    10.127.127.254,
                ],
            ),
            interfaces: Some(
                {
                    "eth0": NetInterface {
                        mac_address: "ca:f8:92:ee:d2:10",
                        subnets: Some(
                            [
                                NetAddress {
                                    gateway: Some(
                                        10.127.127.254,
                                    ),
                                    ipnet: 10.127.127.18/24,
                                },
                            ],
                        ),
                    },
                },
            ),
        },
    }
[DEBUG netavark::commands::setup] "Setup complete"
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroup path for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 to /libpod_parent/libpod-f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/srv/.podman/storage/overlay/6b9cb0043d7d178bbcbab61291b9c887081f551db2a46ccdcdef299ae61ead2b/merged"
DEBU[0000] Created OCI spec for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 at /srv/.podman/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 -u f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 -r /usr/bin/crun -b /srv/.podman/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata -p /run/containers/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata/pidfile -n adoring_golick --exit-dir /var/run/libpod/exits --full-attach -l k8s-file:/srv/.podman/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/containers/storage/overlay-containers/f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.podman/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg /etc/containers/networks/ --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /srv/.podman/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/srv/.podman/images --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg none --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9
DEBU[0000] Tearing down network namespace at /run/netns/netns-e5ea4075-8414-c6f5-b435-8c1944782164 for container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9
[DEBUG netavark::commands::teardown] "Tearing down.."
[DEBUG netavark::firewall] Forcibly using firewall driver none
[INFO  netavark::firewall] Not using firewall
[INFO  netavark::network::bridge] removing bridge podman0
[DEBUG netavark::commands::teardown] "Teardown complete"
DEBU[0000] Unmounted container "f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9"
DEBU[0000] saving container f9db23ed37c60e43a001c8d6e3bcf3d6475ebaa64e1a7bf03ad5f1fce83142f9 state: no such container
DEBU[0000] ExitCode msg: "crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: invalid argument: oci runtime error"
Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument
DEBU[0000] Shutting down engines
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/srv/.podman/images
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false

runc with DEBUG logging

root@OpenWrt:~# podman --log-level debug run --rm --runtime=runc hello-world
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm --runtime=runc hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /srv/.podman/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /srv/.podman/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /srv/.podman/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/srv/.podman/images
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend none
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime uxc initialization failed: no valid executable found for OCI runtime uxc: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/sbin/runc"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded network podman: &{podman 5ef894788befd4d42498314b6e66282ca730aa2e1e82f9b9597bf4d1725ca074 bridge podman0 2023-02-20 08:56:34.652030952 +0000 UTC [{{{10.127.127.0 ffffff00}} 10.127.127.254 <nil>}] [] false false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello-world (policy: missing)
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/hello-world:latest" ...
DEBU[0000] reference "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Looking up image "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "docker.io/library/hello-world:latest" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "docker.io/library/hello-world:latest" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Looking up image "hello-world" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "localhost/hello-world:latest" ...
DEBU[0000] reference "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage
DEBU[0000] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a)
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] Inspecting image d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 3 for container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046
DEBU[0000] parsed reference into "[overlay@/srv/.podman/storage+/run/containers/storage:overlay.imagestore=/srv/.podman/images,overlay.mountopt=nodev]@d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] exporting opaque data as blob "sha256:d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046"
DEBU[0000] Container "7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046" has work directory "/srv/.podman/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata"
DEBU[0000] Container "7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046" has run directory "/run/containers/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating!        PID=5877
DEBU[0000] Enabling signal proxying
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] Made network namespace at /run/netns/netns-f0ee6729-410f-e1d5-d185-578565a02064 for container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046
DEBU[0000] overlay: mount_data=lowerdir=/srv/.podman/storage/overlay/l/LZHX62KQRI3NH3KXXGKPFC2BIO,upperdir=/srv/.podman/storage/overlay/cad85d8adfb52181f045b5dfcf3296e6007c793939aa1f2fddad3907bead89c6/diff,workdir=/srv/.podman/storage/overlay/cad85d8adfb52181f045b5dfcf3296e6007c793939aa1f2fddad3907bead89c6/work,nodev,volatile
DEBU[0000] Mounted container "7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046" at "/srv/.podman/storage/overlay/cad85d8adfb52181f045b5dfcf3296e6007c793939aa1f2fddad3907bead89c6/merged"
DEBU[0000] Created root filesystem for container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 at /srv/.podman/storage/overlay/cad85d8adfb52181f045b5dfcf3296e6007c793939aa1f2fddad3907bead89c6/merged
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[DEBUG netavark::firewall] Forcibly using firewall driver none
[INFO  netavark::firewall] Not using firewall
[DEBUG netavark::network::bridge] Setup network podman
[DEBUG netavark::network::bridge] Container interface name: eth0 with IP addresses [10.127.127.19/24]
[DEBUG netavark::network::bridge] Bridge name: podman0 with IP addresses [10.127.127.254/24]
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth0/autoconf to 0
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth0/arp_notify to 1
[INFO  netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 10.127.127.254, metric 100)
[DEBUG netavark::dns::aardvark] Spawning aardvark server
[DEBUG netavark::dns::aardvark] start aardvark-dns: ["/usr/lib/podman/aardvark-dns", "--config", "/run/containers/networks/aardvark-dns", "-p", "53", "run"]
[DEBUG netavark::commands::setup] {
        "podman": StatusBlock {
            dns_search_domains: Some(
                [
                    "dns.podman",
                ],
            ),
            dns_server_ips: Some(
                [
                    10.127.127.254,
                ],
            ),
            interfaces: Some(
                {
                    "eth0": NetInterface {
                        mac_address: "46:20:6f:d6:44:f2",
                        subnets: Some(
                            [
                                NetAddress {
                                    gateway: Some(
                                        10.127.127.254,
                                    ),
                                    ipnet: 10.127.127.19/24,
                                },
                            ],
                        ),
                    },
                },
            ),
        },
    }
[DEBUG netavark::commands::setup] "Setup complete"
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroup path for container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 to /libpod_parent/libpod-7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/srv/.podman/storage/overlay/cad85d8adfb52181f045b5dfcf3296e6007c793939aa1f2fddad3907bead89c6/merged"
DEBU[0000] Created OCI spec for container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 at /srv/.podman/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 -u 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 -r /usr/sbin/runc -b /srv/.podman/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata -p /run/containers/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata/pidfile -n tender_chatterjee --exit-dir /var/run/libpod/exits --full-attach -l k8s-file:/srv/.podman/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/containers/storage/overlay-containers/7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /srv/.podman/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg /etc/containers/networks/ --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /srv/.podman/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/srv/.podman/images --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg none --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
DEBU[0000] Received: 6064
INFO[0000] Got Conmon PID as 5983
DEBU[0000] Created container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 in OCI runtime
DEBU[0000] Adding nameserver(s) from network status of '["10.127.127.254"]'
DEBU[0000] Adding search domain(s) from network status of '["dns.podman"]'
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
DEBU[0000] Attaching to container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046
DEBU[0000] Starting container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 with command [/hello]
DEBU[0000] Started container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046
DEBU[0000] Notify sent successfully

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

DEBU[0001] Checking if container 7a0a1433a60463c18e1eb734af9bcf4fed2ab357913d5f94898eacd715644046 should restart
DEBU[0001] Called run.PersistentPostRunE(podman --log-level debug run --rm --runtime=runc hello-world)
DEBU[0001] Shutting down engines

giuseppe · 2024-02-29T21:59:52Z

giuseppe
Feb 29, 2024
Maintainer

what is your current cgroup (cat /proc/self/cgroup)?

0 replies

djarbz · 2024-03-01T01:51:26Z

djarbz
Mar 1, 2024
Author

root@OpenWrt:~# cat /proc/self/cgroup
0::/services/dropbear/instance1

0 replies

giuseppe · 2024-03-01T08:57:13Z

giuseppe
Mar 1, 2024
Maintainer

The real error is:

Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument

Some cgroup controllers might be missing in the kernel, can you show the output of cat /sys/fs/cgroup/cgroup.controllers?

If the pids controller is missing, then you will need to modify the configuration in containers.conf and drop the pids_limit.

0 replies

djarbz · 2024-03-01T11:30:28Z

djarbz
Mar 1, 2024
Author

Thanks for looking into this with me.

Looks like pids is there...

root@OpenWrt:~# cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory pids rdma

0 replies

giuseppe · 2024-03-01T11:48:12Z

giuseppe
Mar 1, 2024
Maintainer

To check what controller is failing, you can try:

# echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
# echo +cpu > /sys/fs/cgroup/cgroup.subtree_control
# echo +io > /sys/fs/cgroup/cgroup.subtree_control
# echo +memory > /sys/fs/cgroup/cgroup.subtree_control
# echo +pids > /sys/fs/cgroup/cgroup.subtree_control
# echo +rdma > /sys/fs/cgroup/cgroup.subtree_control

where does it fail?

If it is the CPU controller, a reason could be that you processes running with the realtime scheduler. Is this the case?

In the meanwhile, I've opened a PR for crun to make clearer what it is trying to enable: containers/crun#1430

0 replies

djarbz · 2024-03-01T12:10:16Z

djarbz
Mar 1, 2024
Author

Looks like it doesn't like cpuset.

root@OpenWrt:~# echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
ash: write error: Invalid argument
root@OpenWrt:~# echo +cpu > /sys/fs/cgroup/cgroup.subtree_control
root@OpenWrt:~# echo +io > /sys/fs/cgroup/cgroup.subtree_control
root@OpenWrt:~# echo +memory > /sys/fs/cgroup/cgroup.subtree_control
root@OpenWrt:~# echo +pids > /sys/fs/cgroup/cgroup.subtree_control
root@OpenWrt:~# echo +rdma > /sys/fs/cgroup/cgroup.subtree_control
root@OpenWrt:~#

0 replies

giuseppe · 2024-03-01T12:27:55Z

giuseppe
Mar 1, 2024
Maintainer

that is some weird interaction with kthreads I guess. In any case, it is not something we control from Podman/crun, so I am converting this issue to a discussion

1 reply

djarbz Mar 1, 2024
Author

Thanks for the help, do you have advice on where I should turn to next for assistance?

Can I safely ignore that warning if containers run with runc? I'm not sure what the consequences of cpuset are.

VergilGao · 2024-04-03T05:40:04Z

VergilGao
Apr 3, 2024

got the same error in openwrt 23.05.3

root@OpenWrt:~/app# echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
ash: write error: Invalid argument

here is the debug output

root@OpenWrt:~/app# podman --log-level debug run --rm -it busybox
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm -it busybox)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /root/podman/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /root/podman/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /root/podman/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend none
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime uxc initialization failed: no valid executable found for OCI runtime uxc: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Successfully loaded network podman: &{podman 5ef894788befd4d42498314b6e66282ca730aa2e1e82f9b9597bf4d1725ca074 bridge podman0 2023-02-20 08:56:34.652030952 +0000 UTC [{{{192.168.128.0 ffffff00}} 192.168.128.254 <nil>}] [] false false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image busybox (policy: missing)
DEBU[0000] Looking up image "busybox" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/busybox:latest" ...
DEBU[0000] reference "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]localhost/busybox:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/busybox:latest" ...
DEBU[0000] reference "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]docker.io/library/busybox:latest" does not resolve to an image ID
DEBU[0000] Trying "registry.fedoraproject.org/busybox:latest" ...
DEBU[0000] reference "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]registry.fedoraproject.org/busybox:latest" does not resolve to an image ID
DEBU[0000] Trying "registry.access.redhat.com/busybox:latest" ...
DEBU[0000] reference "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]registry.access.redhat.com/busybox:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/busybox:latest" ...
DEBU[0000] reference "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]docker.io/library/busybox:latest" does not resolve to an image ID
DEBU[0000] Trying "busybox" ...
✔ docker.io/library/busybox:latest
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0001] Attempting to pull candidate docker.io/library/busybox:latest for busybox
DEBU[0001] parsed reference into "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]docker.io/library/busybox:latest"
Trying to pull docker.io/library/busybox:latest...
DEBU[0001] Copying source image //busybox:latest to destination image [overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]docker.io/library/busybox:latest
DEBU[0001] Using registries.d directory /etc/containers/registries.d
DEBU[0001] Trying to access "docker.io/library/busybox:latest"
DEBU[0001] No credentials matching docker.io/library/busybox found in /run/containers/0/auth.json
DEBU[0001] No credentials matching docker.io/library/busybox found in /root/.config/containers/auth.json
DEBU[0001] No credentials matching docker.io/library/busybox found in /root/.docker/config.json
DEBU[0001] No credentials matching docker.io/library/busybox found in /root/.dockercfg
DEBU[0001] No credentials for docker.io/library/busybox found
DEBU[0001]  No signature storage configuration found for docker.io/library/busybox:latest, using built-in default file:///var/lib/containers/sigstore
DEBU[0001] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0001] GET https://registry-1.docker.io/v2/
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0002] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fbusybox%3Apull&service=registry.docker.io
DEBU[0003] GET https://registry-1.docker.io/v2/library/busybox/manifests/latest
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0004] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0004] Source is a manifest list; copying (only) instance sha256:4be429a5fbb2e71ae7958bfa558bc637cf3a61baf40a708cb8fff532b39e52d0 for current system
DEBU[0004] GET https://registry-1.docker.io/v2/library/busybox/manifests/sha256:4be429a5fbb2e71ae7958bfa558bc637cf3a61baf40a708cb8fff532b39e52d0
DEBU[0004] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0004] IsRunningImageAllowed for image docker:docker.io/library/busybox:latest
DEBU[0004]  Using default policy section
DEBU[0004]  Requirement 0: allowed
DEBU[0004] Overall: allowed
DEBU[0004] Downloading /v2/library/busybox/blobs/sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab

DEBU[0004] GET https://registry-1.docker.io/v2/library/busybox/blobs/sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab
Getting image source signatures
DEBU[0005] Reading /var/lib/containers/sigstore/library/busybox@sha256=4be429a5fbb2e71ae7958bfa558bc637cf3a61baf40a708cb8fff532b39e52d0/signature-1
DEBU[0005] Not looking for sigstore attachments: disabled by configuration
DEBU[0005] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0005] ... will first try using the original manifest unmodified
DEBU[0005] Checking if we can reuse blob sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true
DEBU[0005] Failed to retrieve partial blob: convert_images not configured
DEBU[0005] Downloading /v2/library/busybox/blobs/sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c

DEBU[0005] GET https://registry-1.docker.io/v2/library/busybox/blobs/sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c
Copying blob 7b2699543f22 [--------------------------------------] 0.0b / 2.1MiB (skipped: 0.0b = 0.00%)
Copying blob 7b2699543f22 [--------------------------------------] 0.0b / 2.1MiB | 0.0 b/s
Copying blob 7b2699543f22 done   |
DEBU[0005] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0005] Check for idmapped mounts support
DEBU[0005] Applying tar in /root/podman/storage/overlay/95c4a60383f7b6eb6f7b8e153a07cd6e896de0476763bef39d0f6cf3400624bd/diff
DEBU[0005] No compression detected
DEBU[0005] Compression change for blob sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab ("application/vnd.oci.image.config.v1+json") not supported
DEBU[0005] Using original blob without modification
Copying config ba5dc23f65 done   |
Writing manifest to image destination
DEBU[0005] setting image creation date to 2023-05-18 22:34:17 +0000 UTC
DEBU[0005] created new image ID "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" with metadata "{}"
DEBU[0005] added name "docker.io/library/busybox:latest" to image "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Pulled candidate docker.io/library/busybox:latest successfully
DEBU[0005] Looking up image "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" in local containers storage
DEBU[0005] Trying "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" ...
DEBU[0005] parsed reference into "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]@ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Found image "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" as "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" in local containers storage
DEBU[0005] Found image "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" as "ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab" in local containers storage ([overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]@ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab)
DEBU[0005] exporting opaque data as blob "sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Looking up image "busybox" in local containers storage
DEBU[0005] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0005] Trying "docker.io/library/busybox:latest" ...
DEBU[0005] parsed reference into "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]@ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage
DEBU[0005] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage ([overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]@ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab)
DEBU[0005] exporting opaque data as blob "sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Inspecting image ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab
DEBU[0005] exporting opaque data as blob "sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Inspecting image ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab
DEBU[0005] Inspecting image ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab
DEBU[0005] Inspecting image ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab
DEBU[0005] using systemd mode: false
DEBU[0005] No hostname set; container's hostname will default to runtime default
DEBU[0005] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0005] Allocated lock 2 for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0005] parsed reference into "[overlay@/root/podman/storage+/run/containers/storage:overlay.mountopt=nodev]@ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] exporting opaque data as blob "sha256:ba5dc23f65d4cc4a4535bce55cf9e63b068eb02946e3422d3587e8ce803b6aab"
DEBU[0005] Created container "f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8"
DEBU[0005] Container "f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8" has work directory "/root/podman/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata"
DEBU[0005] Container "f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8" has run directory "/run/containers/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata"
DEBU[0005] Handling terminal attach
INFO[0005] Received shutdown.Stop(), terminating!        PID=4501
DEBU[0005] Enabling signal proxying
DEBU[0005] Cached value indicated that volatile is being used
DEBU[0005] overlay: mount_data=lowerdir=/root/podman/storage/overlay/l/HNALAAHDUPIEGTNL3RNULGYHII,upperdir=/root/podman/storage/overlay/79dc97e09736aa3d433d6eb511f7f04c0d601b700673a5f9ca348dea02c1df07/diff,workdir=/root/podman/storage/overlay/79dc97e09736aa3d433d6eb511f7f04c0d601b700673a5f9ca348dea02c1df07/work,nodev,volatile
DEBU[0005] Made network namespace at /run/netns/netns-9bd3bab8-69c9-363c-9f00-57a1a6118d9f for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0005] Mounted container "f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8" at "/root/podman/storage/overlay/79dc97e09736aa3d433d6eb511f7f04c0d601b700673a5f9ca348dea02c1df07/merged"
DEBU[0005] Created root filesystem for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 at /root/podman/storage/overlay/79dc97e09736aa3d433d6eb511f7f04c0d601b700673a5f9ca348dea02c1df07/merged
[DEBUG netavark::network::validation] "Validating network namespace..."
                                                                       [DEBUG netavark::commands::setup] "Setting up..."
                                                                                                                       [DEBUG netavark::firewall] Forcibly using firewall driver none
                                                             [INFO  netavark::firewall] Not using firewall
                                                                                                          [DEBUG netavark::network::bridge] Setup network podman
                                        [DEBUG netavark::network::bridge] Container interface name: eth0 with IP addresses [192.168.128.3/24]
                     [DEBUG netavark::network::bridge] Bridge name: podman0 with IP addresses [192.168.128.254/24]
                                                                                                                  [DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
                                                                                 [DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth0/autoconf to 0
                                                                  [DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth0/arp_notify to 1
                                                     [INFO  netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 192.168.128.254, metric 100)
                               [DEBUG netavark::dns::aardvark] Spawning aardvark server
                                                                                       [DEBUG netavark::dns::aardvark] start aardvark-dns: ["/usr/lib/podman/aardvark-dns", "--config", "/run/containers/networks/aardvark-dns", "-p", "53", "run"]
   [DEBUG netavark::commands::setup] {
                                              "podman": StatusBlock {
                                                                                 dns_search_domains: Some(

  [
                       "dns.podman",
                                                    ],
                                                                  ),
                                                                                dns_server_ips: Some(
                                                                                                                     [

                  192.168.128.254,
                                                  ],
                                                                ),
                                                                              interfaces: Some(
                                                                                                               {

            "eth0": NetInterface {
                                                          mac_address: "42:2a:02:05:af:5b",
                                                                                                                   subnets: Some(
                                     [
                                                                      NetAddress {
                                                                                                                      gateway: Some(
                                                    192.168.128.254,
                                                                                                        ),

                      ipnet: 192.168.128.3/24,
                                                                              },
                                                                                                            ],

              ),
                                    },
                                                      },
                                                                    ),
                                                                              },
                                                                                    }
                                                                                     [DEBUG netavark::commands::setup] "Setup complete"
               DEBU[0005] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0005] Setting Cgroup path for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 to /libpod_parent/libpod-f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0005] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0005] Workdir "/" resolved to host path "/root/podman/storage/overlay/79dc97e09736aa3d433d6eb511f7f04c0d601b700673a5f9ca348dea02c1df07/merged"
DEBU[0005] Created OCI spec for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 at /root/podman/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata/config.json
DEBU[0005] /usr/bin/conmon messages will be logged to syslog
DEBU[0005] running conmon: /usr/bin/conmon               args="[--api-version 1 -c f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 -u f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 -r /usr/bin/crun -b /root/podman/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata -p /run/containers/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata/pidfile -n hungry_hamilton --exit-dir /var/run/libpod/exits --full-attach -l k8s-file:/root/podman/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/containers/storage/overlay-containers/f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /root/podman/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg /etc/containers/networks/ --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /root/podman/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg none --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8]"
WARN[0005] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: write /sys/fs/cgroup/cgroup.subtree_control: invalid argument
DEBU[0006] Cleaning up container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0006] Tearing down network namespace at /run/netns/netns-9bd3bab8-69c9-363c-9f00-57a1a6118d9f for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
[DEBUG netavark::commands::teardown] "Tearing down.."
                                                     [DEBUG netavark::firewall] Forcibly using firewall driver none
                                                                                                                   [INFO
  netavark::firewall] Not using firewall
                                        [INFO  netavark::network::bridge] removing bridge podman0
                                                                                                 [DEBUG netavark::commands::teardown] "Teardown complete"
                                 DEBU[0006] Unmounted container "f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8"
DEBU[0006] Removing container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0006] Cleaning up container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0006] Network is already cleaned up, skipping...
DEBU[0006] Container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 storage is already unmounted, skipping...
DEBU[0006] Removing all exec sessions for container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8
DEBU[0006] Container f00b0e5fb2f274dfa8f8f39b2e8d6590c04788a216c963533123cc1d11646ec8 storage is already unmounted, skipping...
DEBU[0006] ExitCode msg: "container create failed (no logs from conmon): conmon bytes \"\": readobjectstart: expect { or n, but found \x00, error found in #0 byte of ...||..., bigger context ...||..."
Error: container creuname -red (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
5.15.1506] Shutting down engines

0 replies

ctrochalakis · 2024-04-03T06:38:58Z

ctrochalakis
Apr 3, 2024

I had the same issue on a NixOS 23.11 system when I installed podman. I rebooted the system and it worked perfectly after that. I didn't dig any further to find the root cause, I am sharing that in case it helps.

0 replies

ash0080 · 2024-09-24T08:22:33Z

ash0080
Sep 24, 2024

Same issue on OpenWrt 23.05, rebooted then worked, rebooted again then die
crun version 1.14.1

root@ImmortalWrt:/etc# podman --log-level=trace run hello-world
INFO[0000] podman filtering at log level trace          
...
[DEBUG netavark::firewall::nft] Removing 4 rules
[DEBUG netavark::firewall::nft] Found chain nv_5ef89478_10_88_0_0_nm16
[DEBUG netavark::firewall::nft] Matched Rule { family: INet, table: "netavark", chain: "NETAVARK-ISOLATION-3", expr: [Match(Match { left: Named(Meta(Meta { key: Oifname })), right: String("podman0"), op: EQ }), Drop(None)], handle: Some(27), index: None, comment: None }
[DEBUG netavark::firewall::nft] Removing 1 isolation rules for network
[DEBUG netavark::commands::teardown] "Teardown complete"
DEBU[0000] Unmounted container "75a48a87601bf3ba8b8b6d6b8f515eb3742207a2a779696e7d6dc5e6325fdb1c" 
DEBU[0000] ExitCode msg: "crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: invalid argument: oci runtime error" 
Error: OCI runtime error: crun: writing file `/sys/fs/cgroup/cgroup.subtree_control`: Invalid argument
DEBU[0000] Shutting down engines

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to add conmon to cgroupfs sandbox cgroup #21901

{{title}}

Replies: 10 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Failed to add conmon to cgroupfs sandbox cgroup #21901

djarbz Feb 29, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Replies: 10 comments · 1 reply

giuseppe Feb 29, 2024 Maintainer

djarbz Mar 1, 2024 Author

giuseppe Mar 1, 2024 Maintainer

djarbz Mar 1, 2024 Author

giuseppe Mar 1, 2024 Maintainer

djarbz Mar 1, 2024 Author

giuseppe Mar 1, 2024 Maintainer

djarbz Mar 1, 2024 Author

VergilGao Apr 3, 2024

ctrochalakis Apr 3, 2024

ash0080 Sep 24, 2024

djarbz
Feb 29, 2024

Replies: 10 comments 1 reply

giuseppe
Feb 29, 2024
Maintainer

djarbz
Mar 1, 2024
Author

giuseppe
Mar 1, 2024
Maintainer

djarbz
Mar 1, 2024
Author

giuseppe
Mar 1, 2024
Maintainer

djarbz
Mar 1, 2024
Author

giuseppe
Mar 1, 2024
Maintainer

djarbz Mar 1, 2024
Author

VergilGao
Apr 3, 2024

ctrochalakis
Apr 3, 2024

ash0080
Sep 24, 2024