image build error via service when not logged in

[johnwc]

Issue Description

We built a ubuntu 22.04 VM to run self hosted azure agent. Installed podman-docker to support building container images. If we are SSH to the machine using the same user account the agent service uses, the images build just fine. If we disconnect from the VM, they fail with Error: default OCI runtime "crun" not found: invalid argument

Steps to reproduce the issue

Steps to reproduce the issue

1. Fresh deploy ubuntu 22.04
2. Install azure agent as service along with podman-docker
3. Kickoff pipeline that builds a container image

Describe the results you received

Error: default OCI runtime "crun" not found: invalid argument`

Describe the results you expected

Container builds

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.10-0ubuntu22.04+obs18.36_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 85.95
    systemPercent: 1.54
    userPercent: 12.51
  cpus: 2
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  freeLocks: 2048
  hostname: azure-agent02
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-1054-kvm
  linkmode: dynamic
  logDriver: journald
  memFree: 94269440
  memTotal: 3989762048
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.14.4-0ubuntu22.04+obs70.17_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.3-0ubuntu22.04+obs13.21_amd64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 41m 34.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: $HOME/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ubuntu/.local/share/containers/storage
  graphRootAllocated: 77851254784
  graphRootUsed: 15920349184
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 88
  runRoot: /tmp/containers-user-1001/containers
  transientStore: false
  volumePath: /home/ubuntu/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[johnwc]

@rhatdan @Luap99 any thoughts?

[Luap99]

This is most likely not a podman issue, something is wrong with your environment then. You can strace podman to see what is doing but the error basically tells you that it cannot find its OCI runtime so something is wrong with your filesystem if the path is must like this.

[johnwc]

What? The path is magically applied to a daemon running in the background if I am logged in as the user, and not if I am not logged in? Please make that make sense.

[Luap99]

I can only tell you what podman is checking, it is up to you to debug your environment to see what is different between ssh and not.

[johnwc]

I am not saying that I am running it via ssh and it works vs not working when running via the daemon service. I am saying that the daemon service starts working if I am logged into the machine. As soon as I log out of the machine, the daemon service starts throwing that error if building an image.

[eriksjolund]

As soon as I log out of the machine, the daemon service starts throwing that error if building an image.

Does the daemon service run as a systemd user service?

If so, then you probably need to enable lingering with the command

sudo loginctl enable-linger username

Systemd user services are usually stopped after the user logs out if lingering has not been enabled.

Update

I read your post again. Aha, the daemon service is still running as you write daemon service starts throwing that error. Anyway, you could always enable lingering and see if it helps.

[johnwc]

That seems to have fixed the issue. Thank you!