"container state improper" in Rootless Containers/Pod After Reboot #10036

GeorgiaM-honestly · 2021-04-14T23:02:02Z

GeorgiaM-honestly
Apr 14, 2021

Hello,

I'm getting the following issue starting a previously working pod with two containers in it. This only started after a kernel upgrade and reboot, in fact, the first kernel upgrade after installing podman.

Trying to start the pod as usual: podman pod start mdb-pwg
Result:

error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 3cab92372d7b7749e4935ccdd66eb67d861880bf4521aa6afdf9693d102f14e3 failed with status 1, stdout="", stderr="failed to open netns \"/proc/2282/ns/net\": failed to Statfs \"/proc/2282/ns/net\": no such file or directory\n"
error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: container state improper
Error: error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: container state improper

I have looked around in all of the usual places, dmesg, system logs and so on and just have not found anything else interesting. The only for-sure thing is that this started immediately after the kernel upgrade and reboot.

I'm at a loss here :)

Searching around the internet hasn't revealed much, so I'll just get to the juicy stuff and give you:

podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 3f3bf835bb5bd523d4b4850825dbcbc8a52c0407'
  cpus: 2
  distribution:
    distribution: alpine
    version: 3.14.0_alpha20210212
  eventLogger: file
  hostname: Alpine-Images
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.29-0-virt
  linkmode: dynamic
  memFree: 3941535744
  memTotal: 4134445056
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.19
      commit: e67a75672412975916dac6b87f8346f870e4b99a
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-2945/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SET
FCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 536866816
  swapTotal: 536866816
  uptime: 35m 1.06s
registries:
  search:
  - docker.io
store:
  configFile: /home/deleted/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.5
        fusermount3 version: 3.10.3
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/deleted/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/podman-run-2945/containers
  volumePath: /home/deleted/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.0
  Built: 1617144371
  BuiltTime: Tue Mar 30 22:46:11 2021
  GitCommit: 9a786166922beba71bd8934b709440def82091dc
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0

Luap99 · 2021-04-15T09:12:00Z

Luap99
Apr 15, 2021
Maintainer

Is your /tmp on a tmpfs?

7 replies

GeorgiaM-honestly Apr 15, 2021
Author

Okay, I do see entries like that in /tmp before rebooting. After rebooting, they are gone. Thus, I try to start up the pod:

podman pod start mdb-pwg

error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: **container state improper**
error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: **container state improper**
Error: error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 3cab92372d7b7749e4935ccdd66eb67d861880bf4521aa6afdf9693d102f14e3 failed with status 1, stdout="", stderr="failed to open netns \"/proc/2282/ns/net\": failed to Statfs \"/proc/2282/ns/net\": permission denied\n"

Here is the output form podman ps -a:

podman ps -a

CONTAINER ID  IMAGE                                                                                                      COMMAND         CREATED       STATUS                   PORTS                                        NAMES
cba050938651  k8s.gcr.io/pause:3.5                                                                                                       5 days ago    Exited (0) 18 hours ago  0.0.0.0:8080->80/tcp, 0.0.0.0:8443->443/tcp  cda52a83ed9e-infra
1a7771ff115c  docker.io/mariadb/server:latest                                                                            mysqld          5 days ago    **Created**                  0.0.0.0:8080->80/tcp, 0.0.0.0:8443->443/tcp  mariadb
eb004d93dde9  docker.io/linuxserver/piwigo:latest                                                                                        5 days ago    Exited (0) 18 hours ago  0.0.0.0:8080->80/tcp, 0.0.0.0:8443->443/tcp  piwigo
3cab92372d7b  quay.io/libpod/rootless-cni-infra@sha256:adf352454666f7ce9ca3e1098448b5ee18f89c4516471ec99447ec9ece917f36  sleep infinity  18 hours ago  Up About a minute ago                                                 rootless-cni-infra

It looks like mariadb container is stuck in the created / configured state?

podman container inspect mariadb:

[
    {
        "Id": "1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9",
        "Created": "2021-04-10T04:03:20.443868647Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "mysqld"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            **"Status": "configured",**
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-04-14T19:32:11.720009959Z",
            "FinishedAt": "2021-04-14T19:34:09.593078691Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "56ef055b7d2c31b2235180b0e4974741da1750d4bbf7a8dfa55d49084502e25d",
        "ImageName": "docker.io/mariadb/server:latest",
        "Rootfs": "",
        "Pod": "cda52a83ed9eb9ed1b0f9b5e65757dd09ac6087b5e2f3e842775c3b49844fc12",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "StaticDir": "/home/deleted/.local/share/containers/storage/overlay-containers/1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9/userdata",
        "OCIConfigPath": "/home/deleted/.local/share/containers/storage/overlay-containers/1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/tmp/podman-run-2945/containers/overlay-containers/1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9/userdata/conmon.pid",
        "Name": "mariadb",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/deleted/.local/share/containers/storage/overlay/ac83605687923b2ac674d956a3eacfe1d2471ba9b227608571338c0bfea8bd5e/diff:/home/deleted/.local/share/containers/storage/overlay/8e49a236dad02d3dd28c1d2ab7a5c2f55139d4033221dd96a922e166b2aa06b9/diff:/home/deleted/.local/share/containers/storage/overlay/3fd4103977d7b82e4f1955a0b83f9b03dd07190526ad2d1063ff65794ceaa21c/diff:/home/deleted/.local/share/containers/storage/overlay/2a889f0d342f4fdd76e13ecd2e6f97723e2e5535816a6c4867bf48b421c1c07c/diff:/home/deleted/.local/share/containers/storage/overlay/0ef78629ac5fb467537b38f356f042133bb2674284c3579458463f5db3698ae5/diff:/home/deleted/.local/share/containers/storage/overlay/876a8fcb45d1464aae00472f7311e41cac3c0e867205742f523c54c457cab9d1/diff:/home/deleted/.local/share/containers/storage/overlay/bdaff751f6a487b2e807f2483b85f8704bb55366a45282c7f69158d564b58e80/diff:/home/deleted/.local/share/containers/storage/overlay/64818edbd6300346c86222ca6a0c4d58679944d2d0856ea7a3bb9eab3c94aa2b/diff:/home/deleted/.local/share/containers/storage/overlay/d9a86197153f68f261561e61a5d460533756fc521958191b1df6dcc80d50c023/diff:/home/deleted/.local/share/containers/storage/overlay/9545fd5187690271085ba79b2acc997dcd3d40ca733e89ff398cc028583a5e78/diff:/home/deleted/.local/share/containers/storage/overlay/b4348bfd84050f05a39ebe017b137d3bde5ce307cc5dd819495c4f362587c4e3/diff:/home/deleted/.local/share/containers/storage/overlay/9dd0b45ebe39e477b2a869802e9b771d991f7341cb79e3446c9ca8231fc35d3f/diff:/home/deleted/.local/share/containers/storage/overlay/c95d2191d7773c6e29188f92922bc9547e1f0b6130e85dfc2f5e4eae13137c7c/diff",
                "UpperDir": "/home/deleted/.local/share/containers/storage/overlay/ea07fc7b2a9b8e4a7f0106eab6fb3841665bb114df2710a84019793b9fe6a03e/diff",
                "WorkDir": "/home/deleted/.local/share/containers/storage/overlay/ea07fc7b2a9b8e4a7f0106eab6fb3841665bb114df2710a84019793b9fe6a03e/work"
            }
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "4c532ec450f8bc24f985f182bd4d2c45a5278843afe78e0bf81102f9ccf27340",
                "Source": "/home/deleted/.local/share/containers/storage/volumes/4c532ec450f8bc24f985f182bd4d2c45a5278843afe78e0bf81102f9ccf27340/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nodev",
                    "exec",
                    "nosuid",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [
            "cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b"
        ],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8443"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8080"
                    }
                ]
            },
            "SandboxKey": "",
            "Networks": {
                "mdb-pwg": {
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "NetworkID": "mdb-pwg",
                    "DriverOpts": null,
                    "IPAMConfig": null,
                    "Links": null
                }
            }
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/home/deleted/.local/share/containers/storage",
            "--runroot",
            "/tmp/podman-run-2945/containers",
            "--log-level",
            "warning",
            "--cgroup-manager",
            "cgroupfs",
            "--tmpdir",
            "/tmp/run-2945/libpod/tmp",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "--events-backend",
            "file",
            "container",
            "cleanup",
            "1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "mdb-pwg",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=podman",
                "MYSQL_ROOT_PASSWORD=d;NNsaskYHUbGm(FF$]exJW.LLsEoSHdm)@4t]G7XiysqQUq;5a%QR2m~si29",
                "MARIADB_MAJOR=10.5",
                "GOSU_VERSION=1.12",
                "GPG_KEYS=199369E5404BD5FC7D2FE43BCBCB082A1BB943DB \t177F4010FE56CA3336300305F1656F24C74CD1D8",
                "HOME=/root",
                "HOSTNAME=mdb-pwg"
            ],
            "Cmd": [
                "mysqld"
            ],
            "Image": "docker.io/mariadb/server:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": "docker-entrypoint.sh",
            "OnBuild": null,
            "Labels": null,
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.ContainerType": "container",
                "io.kubernetes.cri-o.Created": "2021-04-10T04:03:20.443868647Z",
                "io.kubernetes.cri-o.SandboxID": "mdb-pwg",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "podman",
                "run",
                "-dt",
                "--name",
                "mariadb",
                "--pod",
                "mdb-pwg",
                "-e",
                "MYSQL_ROOT_PASSWORD=d;NNsaskYHUbGm(FF$]exJW.LLsEoSHdm)@4t]G7XiysqQUq;5a%QR2m~si29",
                "mariadb/server:latest"
            ],
            "Umask": "0022"
        },
        "HostConfig": {
            "Binds": [
                "4c532ec450f8bc24f985f182bd4d2c45a5278843afe78e0bf81102f9ccf27340:/var/lib/mysql:rprivate,rw,nodev,exec,nosuid,rbind"
            ],
            "CgroupManager": "cgroupfs",
            "CgroupMode": "host",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/home/deleted/.local/share/containers/storage/overlay-containers/1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9/userdata/ctr.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "container:cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [
                "CAP_AUDIT_WRITE",
                "CAP_MKNOD",
                "CAP_NET_RAW"
            ],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "container:cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "container:cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "/libpod_parent/cda52a83ed9eb9ed1b0f9b5e65757dd09ac6087b5e2f3e842775c3b49844fc12",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": [],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]

mheon Apr 15, 2021
Maintainer

This is definitely still an issue with Podman not detecting the restart; container state was not refreshed.

Please run podman info --log-level=debug and locate the line saying DEBU[0000] Using run root ... - the path mentioned in that error message must be removed before the first Podman command after a reboot is run (specifically, the alive file in said directory).

Luap99 Apr 15, 2021
Maintainer

Try to run podman stop -a && podman container cleanup -a

GeorgiaM-honestly Apr 15, 2021
Author

This is definitely still an issue with Podman not detecting the restart; container state was not refreshed.

Please run podman info --log-level=debug and locate the line saying DEBU[0000] Using run root ... - the path mentioned in that error message must be removed before the first Podman command after a reboot is run (specifically, the alive file in said directory).

Okay! Giving this a run, after reboot, there is nothing in /tmp. I then run the following command to find the line you are talking about:

podman info --log-level=debug 1> /dev/null 2> podman-info-debug-STDERR.txt ; grep "Using run root" podman-info-debug-STDERR.txt

Which results in:

time="2021-04-15T17:31:24Z" level=debug msg="Using run root /tmp/podman-run-2945/containers"
time="2021-04-15T17:31:24Z" level=debug msg="Using run root /tmp/podman-run-2945/containers"

These indeed now are in /tmp:

drwx------    3 deleted deleted      4096 Apr 15 17:31 containers-user-2945
drwx------    4 deleted deleted      4096 Apr 15 17:31 podman-run-2945
drwx------    3 deleted deleted      4096 Apr 15 17:31 run-2945

As is the alive file:

-rw-r--r--    1 deleted deleted         0 Apr 15 17:31 /tmp/run-2945/libpod/tmp/alive
-rw-r--r--    1 deleted deleted         0 Apr 15 17:31 /tmp/run-2945/libpod/tmp/alive.lck

All of these are not present after a reboot however they are after running the podman info command. Removing all of these and then trying to start the pod at this point returns:

podman pod start mdb-pwg

error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: container state improper
error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 3cab92372d7b7749e4935ccdd66eb67d861880bf4521aa6afdf9693d102f14e3 failed with status 1, stdout="", stderr="failed to open netns \"/proc/2282/ns/net\": failed to Statfs \"/proc/2282/ns/net\": no such file or directory\n"
Error: error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: container state improper

And the files are back in /tmp/. The tmp files have 2945 in them, however, the above error mentions 2282. Are these both PIDs? If so, the 2282 has stayed the same since the beginning of these errors so I don't really understand why there is a reference to 2282 and 2945; Should they match up? I'm assuming I'm missing something here.

GeorgiaM-honestly Apr 15, 2021
Author

podman stop -a && podman container cleanup -a

Hello, thanks a lot for the response and your time.

After rebooting (Note: On this system, /tmp/ is cleared out at boot) and doing so and trying to start the pod:

podman pod start mdb-pwg

error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 3cab92372d7b7749e4935ccdd66eb67d861880bf4521aa6afdf9693d102f14e3 failed with status 1, stdout="", stderr="error switching to ns /proc/2282/ns/net: Error switching to ns /proc/2282/ns/net: operation not permitted\n"
error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: container state improper
Error: error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: container state improper

As I imagine is expected, but just to be sure I'll show it here after the fact:

drwx------    3 deleted deleted      4096 Apr 15 17:46 containers-user-2945
drwx------    6 deleted deleted      4096 Apr 15 17:46 podman-run-2945
drwx------    3 deleted deleted      4096 Apr 15 17:46 run-2945

Adding this from my former comment:

The tmp files have 2945 in them, however, the above error mentions 2282. Are these both PIDs? If so, the 2282 has stayed the same since the beginning of these errors so I don't really understand why there is a reference to 2282 and 2945; Should they match up? I'm assuming I'm missing something here.

GeorgiaM-honestly · 2021-04-16T02:42:43Z

GeorgiaM-honestly
Apr 16, 2021
Author

Aaaaand the issue just went away on its own... I was attempting to use "backups(?)" created before these issues with podman export, by creating a new pod, new network, and importing the exports as images and then deploying containers. For giggles I tried to start up the pod with the containers having the issues above, and it just... worked. No more errors.

Am I nuts? Or was "something" stale somewhere or whatever? How can I be sure this doesn't happen again? Is my method of backups sound? I am also backing up the volumes.

mind is boggling

0 replies

mheon · 2021-04-16T13:19:44Z

mheon
Apr 16, 2021
Maintainer

Evidence points to this not being a refresh issue, as I initially suspected. I'm now pivoting to the theory this is an EPERM joining the rootless CNI network namespace. It looks like this is an upgrade from an earlier Podman, so we have the new rootless CNI code interacting with the old CNI infra container, which I suspect is the cause.

3 replies

mheon Apr 16, 2021
Maintainer

@Luap99 Should we investigate making Podman break dependencies on the rootless CNI infra container, maybe as part of podman system restore?

Luap99 Apr 16, 2021
Maintainer

According to the podman version it is still v3.1.0 so none of my new rootless networking code is in there. Also the rootless-cni-container was never added as dependency for the containers. This all happened without libpod dependencies. I tested the migration and it should work fine.
I think the issue here is that somehow /proc/2282/ns/net as net namespace got stucked in the container state and was not cleanup after reboot. Since process 2282 no longer existed after the reboot but podman tried to use it it failed to start the container.

mheon Apr 16, 2021
Maintainer

Damn, sorry, I'm confusing my versions.

GeorgiaM-honestly · 2021-04-16T13:45:42Z

GeorgiaM-honestly
Apr 16, 2021
Author

FYI, I am available on Freenode as the nick Fieldy if there's anything I can do to assist. Thanks for all your work!

1 reply

GeorgiaM-honestly Apr 18, 2021
Author

Question:

The issue "went away" and things are running well. My concern is: What do I do if this happens again? I am currently keeping backups like so:

podman export of each of the two containers within the pod
Backups of any volumes at the same time
Multiple backups of both of those

Will this be enough to, if the issue returns, get things going again in the following manner:

Create new network
Create new pod
Import each of the backed up containers as images
Deploy each of the containers with a different name into the new pod

Are these backup / restore approaches good enough?

GeorgiaM-honestly · 2021-04-18T17:42:10Z

GeorgiaM-honestly
Apr 18, 2021
Author

Well, the issue has returned, and like the other time, after a reboot. I have done other reboots between then and now and have not had issues until then.

I have made attempts to perform backups and develop a restoration workflow in case this happens again. Can someone please review the above question about that workflow? #10036 (reply in thread)

podman pod start mdb-pwg

cda52a83ed9eb9ed1b0f9b5e65757dd09ac6087b5e2f3e842775c3b49844fc12
error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 8180929742e737a6e6f6fd9b59c3108f6d5861c5452f56c203f57b850bbeb7c1 failed with status 1, stdout="", stderr="failed to open netns \"/proc/20698/ns/net\": failed to Statfs \"/proc/20698/ns/net\": no such file or directory\n"
error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: container state improper
Error: error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: container state improper

podman stop -a && podman container cleanup -a 

8180929742e737a6e6f6fd9b59c3108f6d5861c5452f56c203f57b850bbeb7c1                                                            cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b
eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51
1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9
1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9
8180929742e737a6e6f6fd9b59c3108f6d5861c5452f56c203f57b850bbeb7c1
cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b
eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51

podman pod start mdb-pwg

error starting container cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b: command rootless-cni-infra [alloc cba050938651e5a389bd0ba1f9c277bf5e27e809f755f5f73b89460c3d64e54b mdb-pwg mdb-pwg   ] in container 8180929742e737a6e6f6fd9b59c3108f6d5861c5452f56c203f57b850bbeb7c1 failed with status 1, stdout="", stderr="failed to open netns \"/proc/20698/ns/net\": failed to Statfs \"/proc/20698/ns/net\": no such file or directory\n"
error starting container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9: a dependency of container 1a7771ff115c1586ff6ec1fd19f570e9f4ce54829aee8f988ae590523fdc75a9 failed to start: container state improper
Error: error starting container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51: a dependency of container eb004d93dde9f518c6c49277f4da261d05e57c2b3ecfeff524f7b2de8a247a51 failed to start: container state improper

podman info loglevel=debug STDOUT:

host:
  arch: amd64
  buildahVersion: 1.20.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 3f3bf835bb5bd523d4b4850825dbcbc8a52c0407'
  cpus: 2
  distribution:
    distribution: alpine
    version: 3.14.0_alpha20210212
  eventLogger: file
  hostname: Alpine-Images
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.29-0-virt
  linkmode: dynamic
  memFree: 3940438016
  memTotal: 4134453248
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.19
      commit: e67a75672412975916dac6b87f8346f870e4b99a
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-2945/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 536866816
  swapTotal: 536866816
  uptime: 14m 32.46s
registries:
  search:
  - docker.io
store:
  configFile: /home/deleted/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.5
        fusermount3 version: 3.10.3
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/deleted/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/podman-run-2945/containers
  volumePath: /home/deleted/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.0
  Built: 1617144371
  BuiltTime: Tue Mar 30 22:46:11 2021
  GitCommit: 9a786166922beba71bd8934b709440def82091dc
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0

podman info loglevel=debug STDERR:

time="2021-04-18T17:37:21Z" level=info msg="podman filtering at log level debug"
time="2021-04-18T17:37:21Z" level=debug msg="Called info.PersistentPreRunE(podman info --log-level=debug)"
time="2021-04-18T17:37:21Z" level=debug msg="Reading configuration file \"/usr/share/containers/containers.conf\""
time="2021-04-18T17:37:21Z" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.35.4 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/run-2945/libpod/tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T17:37:21Z" level=debug msg="Reading configuration file \"/etc/containers/containers.conf\""
time="2021-04-18T17:37:21Z" level=debug msg="Merged system config \"/etc/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.35.4 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/run-2945/libpod/tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T17:37:21Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2021-04-18T17:37:21Z" level=debug msg="Initializing boltdb state at /home/deleted/.local/share/containers/storage/libpod/bolt_state.db"
time="2021-04-18T17:37:21Z" level=debug msg="Using graph driver overlay"
time="2021-04-18T17:37:21Z" level=debug msg="Using graph root /home/deleted/.local/share/containers/storage"
time="2021-04-18T17:37:21Z" level=debug msg="Using run root /tmp/podman-run-2945/containers"
time="2021-04-18T17:37:21Z" level=debug msg="Using static dir /home/deleted/.local/share/containers/storage/libpod"
time="2021-04-18T17:37:21Z" level=debug msg="Using tmp dir /tmp/run-2945/libpod/tmp"
time="2021-04-18T17:37:21Z" level=debug msg="Using volume path /home/deleted/.local/share/containers/storage/volumes"
time="2021-04-18T17:37:21Z" level=debug msg="Set libpod namespace to \"\""
time="2021-04-18T17:37:21Z" level=debug msg="Not configuring container store"
time="2021-04-18T17:37:21Z" level=debug msg="Initializing event backend file"
time="2021-04-18T17:37:21Z" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
time="2021-04-18T17:37:21Z" level=debug msg="using runtime \"/usr/bin/crun\""
time="2021-04-18T17:37:21Z" level=debug msg="using runtime \"/usr/bin/runc\""
time="2021-04-18T17:37:21Z" level=info msg="Setting parallel job count to 7"
time="2021-04-18T17:37:21Z" level=info msg="podman filtering at log level debug"
time="2021-04-18T17:37:21Z" level=debug msg="Called info.PersistentPreRunE(podman info --log-level=debug)"
time="2021-04-18T17:37:21Z" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T17:37:21Z" level=debug msg="Reading configuration file \"/usr/share/containers/containers.conf\""
time="2021-04-18T17:37:21Z" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.35.4 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/podman-run-2945/libpod/tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/podman-run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T17:37:21Z" level=debug msg="Reading configuration file \"/etc/containers/containers.conf\""
time="2021-04-18T17:37:21Z" level=debug msg="Merged system config \"/etc/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.35.4 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/podman-run-2945/libpod/tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/podman-run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T17:37:21Z" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T17:37:21Z" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2021-04-18T17:37:21Z" level=debug msg="Initializing boltdb state at /home/deleted/.local/share/containers/storage/libpod/bolt_state.db"
time="2021-04-18T17:37:21Z" level=debug msg="Overriding tmp dir \"/tmp/podman-run-2945/libpod/tmp\" with \"/tmp/run-2945/libpod/tmp\" from database"
time="2021-04-18T17:37:21Z" level=debug msg="Using graph driver overlay"
time="2021-04-18T17:37:21Z" level=debug msg="Using graph root /home/deleted/.local/share/containers/storage"
time="2021-04-18T17:37:21Z" level=debug msg="Using run root /tmp/podman-run-2945/containers"
time="2021-04-18T17:37:21Z" level=debug msg="Using static dir /home/deleted/.local/share/containers/storage/libpod"
time="2021-04-18T17:37:21Z" level=debug msg="Using tmp dir /tmp/run-2945/libpod/tmp"
time="2021-04-18T17:37:21Z" level=debug msg="Using volume path /home/deleted/.local/share/containers/storage/volumes"
time="2021-04-18T17:37:21Z" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T17:37:21Z" level=debug msg="Set libpod namespace to \"\""
time="2021-04-18T17:37:21Z" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2021-04-18T17:37:21Z" level=debug msg="overlay: mount_program=/usr/bin/fuse-overlayfs"
time="2021-04-18T17:37:21Z" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false"
time="2021-04-18T17:37:21Z" level=debug msg="Initializing event backend file"
time="2021-04-18T17:37:21Z" level=debug msg="using runtime \"/usr/bin/crun\""
time="2021-04-18T17:37:21Z" level=debug msg="using runtime \"/usr/bin/runc\""
time="2021-04-18T17:37:21Z" level=info msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument"
time="2021-04-18T17:37:21Z" level=info msg="Setting parallel job count to 7"
time="2021-04-18T17:37:21Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2021-04-18T17:37:21Z" level=debug msg="Called info.PersistentPostRunE(podman info --log-level=debug)"

0 replies

GeorgiaM-honestly · 2021-04-18T20:56:21Z

GeorgiaM-honestly
Apr 18, 2021
Author

... Further, I tried to restore from backups into a new pod with a new network etc... And it's happened again. Is this a confirmed bug or something I'm doing wrong? This is keeping me from running a site and if I can't even get restorations to work, it's a roadblock.

I'm on podman 3.1.0. I see that 3.1.1 is now available on alpine. Should I give that a try?

podman pod start mdb-pwg-2

error starting container 72e4388d46d8a3595f8c15f6c4e5b4d4bb653bd881a2676432bc6c7a078ddc40: command rootless-cni-infra [alloc 72e4388d46d8a3595f8c15f6c4e5b4d4bb653bd881a2676432bc6c7a078ddc40 mdb-pwg-2 mdb-pwg-2   ] in container 51a6c69edd952c6699406d0c7f340f71cbfc645d695b9e07537d290b952d3555 failed with status 1, stdout="", stderr="failed to open netns \"/proc/11405/ns/net\": failed to Statfs \"/proc/11405/ns/net\": no such file or directory\n"
error starting container 231a07c8eb1c7f83f91ac585cba8e36d85dc8d528ef72930e9463a0ce7648787: a dependency of container 231a07c8eb1c7f83f91ac585cba8e36d85dc8d528ef72930e9463a0ce7648787 failed to start: container state improper
Error: error starting container f240ff6f533bba59ead56e0156bed19da653039e5fd8d608370f7afc67baf123: a dependency of container f240ff6f533bba59ead56e0156bed19da653039e5fd8d608370f7afc67baf123 failed to start: container state improper

0 replies

GeorgiaM-honestly · 2021-04-18T22:05:54Z

GeorgiaM-honestly
Apr 18, 2021
Author

I have time on my hands so tl;dr I replicated this in a virtual machine with 3.1.1. TL;DR: The issue still eventually happens.

podman pod start mdb-pwg 

error starting container cc07f3805a68d8d9d173860455dabc905b4de43f501f8ae7bb48aa642beb0251: command rootless-cni-infra [alloc cc07f3805a68d8d9d173860455dabc905b4de43f501f8ae7bb48aa642beb0251 mdb-pwg mdb-pwg   ] in container 0d0bb0dc1b85ae34a405ac6af0bee820417659979aecd5549622ea2aaf89b6a7 failed with status 1, stdout="", stderr="failed to open netns \"/proc/5675/ns/net\": failed to Statfs \"/proc/5675/ns/net\": no such file or directory\n"
error starting container 49ead99124856a05c57a40b8f142b1085a3629d02d88edfa871137c150425620: a dependency of container 49ead99124856a05c57a40b8f142b1085a3629d02d88edfa871137c150425620 failed to start: container state improper
Error: error starting container a77f562393b5b1e7c5e65a867bde15a4742d9255d6f165e3bfc1fa058c693f46: a dependency of container a77f562393b5b1e7c5e65a867bde15a4742d9255d6f165e3bfc1fa058c693f46 failed to start: container state improper

podman container cleanup -a

ERRO[0000] Error cleaning up container: failed to cleanup container 0d0bb0dc1b85ae34a405ac6af0bee820417659979aecd5549622ea2aaf89b6a7: container 0d0bb0dc1b85ae34a405ac6af0bee820417659979aecd5549622ea2aaf89b6a7 is running or paused, refusing to clean up: container state improper 
49ead99124856a05c57a40b8f142b1085a3629d02d88edfa871137c150425620
a77f562393b5b1e7c5e65a867bde15a4742d9255d6f165e3bfc1fa058c693f46
cc07f3805a68d8d9d173860455dabc905b4de43f501f8ae7bb48aa642beb0251
Error: failed to cleanup container 0d0bb0dc1b85ae34a405ac6af0bee820417659979aecd5549622ea2aaf89b6a7: container 0d0bb0dc1b85ae34a405ac6af0bee820417659979aecd5549622ea2aaf89b6a7 is running or paused, refusing to clean up: container state improper

podman info --log-level=debug STDOUT:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 3f3bf835bb5bd523d4b4850825dbcbc8a52c0407'
  cpus: 16
  distribution:
    distribution: alpine
    version: 3.14.0_alpha20210212
  eventLogger: file
  hostname: piwigo-test
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2945
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.30-0-virt
  linkmode: dynamic
  memFree: 3799400448
  memTotal: 4102262784
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.19
      commit: e67a75672412975916dac6b87f8346f870e4b99a
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-2945/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 3m 11.24s
registries:
  search:
  - docker.io
store:
  configFile: /home/deleted/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.5
        fusermount3 version: 3.10.3
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/deleted/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/podman-run-2945/containers
  volumePath: /home/deleted/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.1
  Built: 1618615474
  BuiltTime: Fri Apr 16 19:24:34 2021
  GitCommit: 2eb6dbcb8ec86a5a9cea15612c41ad11cb6e95a4
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.1.1

podman info --log-level=debug STDERR:

time="2021-04-18T18:01:57-04:00" level=info msg="podman filtering at log level debug"
time="2021-04-18T18:01:57-04:00" level=debug msg="Called info.PersistentPreRunE(podman info --log-level=debug)"
time="2021-04-18T18:01:57-04:00" level=debug msg="Reading configuration file \"/usr/share/containers/containers.conf\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.36.0
 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.pi
ng_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:
/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private S
eccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH
=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/c
onmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/run-2945/libpod/tmp/eve
nts/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: Infr
aImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRunt
ime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbi
n/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local
/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: S
erviceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRo
otSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNot
ify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/vo
lumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T18:01:57-04:00" level=debug msg="Reading configuration file \"/etc/containers/containers.conf\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Merged system config \"/etc/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.36.0 Annot
ations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_gro
up_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/b
in:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private Seccomp
Profile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/
local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon 
/usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/run-2945/libpod/tmp/events/ev
ents.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: InfraImage
:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:cr
un OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata
-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/r
unc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: Service
Destinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:
false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:fa
lse StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumesolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.d}}"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Initializing boltdb state at /home/deleted/.local/share/containers/storage/libpod/bolt_state.db"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using graph driver overlay"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using graph root /home/deleted/.local/share/containers/storage"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using run root /tmp/podman-run-2945/containers"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using static dir /home/deleted/.local/share/containers/storage/libpod"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using tmp dir /tmp/run-2945/libpod/tmp"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using volume path /home/deleted/.local/share/containers/storage/volumes"
time="2021-04-18T18:01:57-04:00" level=debug msg="Set libpod namespace to \"\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Not configuring container store"
time="2021-04-18T18:01:57-04:00" level=debug msg="Initializing event backend file"
time="2021-04-18T18:01:57-04:00" level=debug msg="configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2021-04-18T18:01:57-04:00" level=info msg="Setting parallel job count to 49"
time="2021-04-18T18:01:57-04:00" level=info msg="podman filtering at log level debug"
time="2021-04-18T18:01:57-04:00" level=debug msg="Called info.PersistentPreRunE(podman info --log-level=debug)"
time="2021-04-18T18:01:57-04:00" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T18:01:57-04:00" level=debug msg="Reading configuration file \"/usr/share/containers/containers.conf\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.36.0
 Annotations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.pi
ng_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:
/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private S
eccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH
=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/c
onmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/podman-run-2945/libpod/
tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraComman
d: InfraImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 
OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /
usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /us
r/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveSer
vice: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConf
igRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.jso
n SDNotify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/podman-run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/storage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cn
i/net.d}}"
time="2021-04-18T18:01:57-04:00" level=debug msg="Reading configuration file \"/etc/containers/containers.conf\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Merged system config \"/etc/containers/containers.conf\": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.36.0 Annot
ations:[] CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_gro
up_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/b
in:/sbin:/bin TERM=xterm] EnvHost:false HTTPProxy:true Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private Seccomp
Profile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:cgroupfs ConmonEnvVars:[PATH=/usr/
local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon 
/usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/tmp/podman-run-2945/libpod/tmp/ev
ents/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: InfraCommand: Inf
raImage:k8s.gcr.io/pause:3.5 InitPath:/usr/libexec/podman/catatonit LockType:shm MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRun
time:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sb
in/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/loca
l/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: 
ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunR
ootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNo
tify:false StateType:3 StaticDir:/home/deleted/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/tmp/podman-run-2945/libpod/tmp VolumePath:/home/deleted/.local/share/containers/st
orage/volumes VolumePlugins:map[]} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/deleted/.config/cni/net.
d}}"
time="2021-04-18T18:01:57-04:00" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using conmon: \"/usr/bin/conmon\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Initializing boltdb state at /home/deleted/.local/share/containers/storage/libpod/bolt_state.db"
time="2021-04-18T18:01:57-04:00" level=debug msg="Overriding tmp dir \"/tmp/podman-run-2945/libpod/tmp\" with \"/tmp/run-2945/libpod/tmp\" from database"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using graph driver overlay"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using graph root /home/deleted/.local/share/containers/storage"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using run root /tmp/podman-run-2945/containers"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using static dir /home/deleted/.local/share/containers/storage/libpod"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using tmp dir /tmp/run-2945/libpod/tmp"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using volume path /home/deleted/.local/share/containers/storage/volumes"
time="2021-04-18T18:01:57-04:00" level=debug msg="overlay storage already configured with a mount-program"
time="2021-04-18T18:01:57-04:00" level=debug msg="Set libpod namespace to \"\""
time="2021-04-18T18:01:57-04:00" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2021-04-18T18:01:57-04:00" level=debug msg="overlay: mount_program=/usr/bin/fuse-overlayfs"
time="2021-04-18T18:01:57-04:00" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false"
time="2021-04-18T18:01:57-04:00" level=debug msg="Initializing event backend file"
time="2021-04-18T18:01:57-04:00" level=debug msg="configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2021-04-18T18:01:57-04:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2021-04-18T18:01:57-04:00" level=info msg="Setting parallel job count to 49"
time="2021-04-18T18:01:57-04:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2021-04-18T18:01:57-04:00" level=debug msg="Called info.PersistentPostRunE(podman info --log-level=debug)"

3 replies

Luap99 Apr 19, 2021
Maintainer

OK, I think I know what is going on. After the reboot check if the rootless cni container is still present (podman ps -a | grep rootless-cni-infra). I think it will be there, remove it with podman rm -f rootless-cni-infra. Afterwards you should be able to start your containers/pods. The problems happens when the container shutdown order is not correct. The rootless-cni-infra container should not be stopped manually. However if you shutdown/reboot there is no guarantee that the rootless-cni-infra container is stopped last.
You have two workarounds:

Before you shutdown/reboot stopped all your containers by name. When all containers are stopped the rootless-cni-infra container will be automatically removed.
After the boot remove the rootless-cni-infra container when present.

Luap99 Apr 19, 2021
Maintainer

The good news is that this should be fixed with podman v3.2 because this version will no longer need the rootless-cni-infra container.

GeorgiaM-honestly Apr 19, 2021
Author

OK, I think I know what is going on. After the reboot check if the rootless cni container is still present (podman ps -a | grep rootless-cni-infra). I think it will be there, remove it with podman rm -f rootless-cni-infra. Afterwards you should be able to start your containers/pods. The problems happens when the container shutdown order is not correct. The rootless-cni-infra container should not be stopped manually. However if you shutdown/reboot there is no guarantee that the rootless-cni-infra container is stopped last.
You have two workarounds:

Before you shutdown/reboot stopped all your containers by name. When all containers are stopped the rootless-cni-infra container will be automatically removed.

After the boot remove the rootless-cni-infra container when present.

I think you are correct. I did these things and rebooted 10++ times and did not have a single issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"container state improper" in Rootless Containers/Pod After Reboot #10036

{{title}}

Replies: 7 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"container state improper" in Rootless Containers/Pod After Reboot #10036

GeorgiaM-honestly Apr 14, 2021

Replies: 7 comments · 14 replies

Luap99 Apr 15, 2021 Maintainer

GeorgiaM-honestly Apr 15, 2021 Author

mheon Apr 15, 2021 Maintainer

Luap99 Apr 15, 2021 Maintainer

GeorgiaM-honestly Apr 15, 2021 Author

GeorgiaM-honestly Apr 15, 2021 Author

GeorgiaM-honestly Apr 16, 2021 Author

mheon Apr 16, 2021 Maintainer

mheon Apr 16, 2021 Maintainer

Luap99 Apr 16, 2021 Maintainer

mheon Apr 16, 2021 Maintainer

GeorgiaM-honestly Apr 16, 2021 Author

GeorgiaM-honestly Apr 18, 2021 Author

GeorgiaM-honestly Apr 18, 2021 Author

GeorgiaM-honestly Apr 18, 2021 Author

GeorgiaM-honestly Apr 18, 2021 Author

Luap99 Apr 19, 2021 Maintainer

Luap99 Apr 19, 2021 Maintainer

GeorgiaM-honestly Apr 19, 2021 Author

GeorgiaM-honestly
Apr 14, 2021

Replies: 7 comments 14 replies

Luap99
Apr 15, 2021
Maintainer

GeorgiaM-honestly Apr 15, 2021
Author

mheon Apr 15, 2021
Maintainer

Luap99 Apr 15, 2021
Maintainer

GeorgiaM-honestly Apr 15, 2021
Author

GeorgiaM-honestly Apr 15, 2021
Author

GeorgiaM-honestly
Apr 16, 2021
Author

mheon
Apr 16, 2021
Maintainer

mheon Apr 16, 2021
Maintainer

Luap99 Apr 16, 2021
Maintainer

mheon Apr 16, 2021
Maintainer

GeorgiaM-honestly
Apr 16, 2021
Author

GeorgiaM-honestly Apr 18, 2021
Author

GeorgiaM-honestly
Apr 18, 2021
Author

GeorgiaM-honestly
Apr 18, 2021
Author

GeorgiaM-honestly
Apr 18, 2021
Author

Luap99 Apr 19, 2021
Maintainer

Luap99 Apr 19, 2021
Maintainer

GeorgiaM-honestly Apr 19, 2021
Author