Run rootless podman network in LXC container

[davispuh]

I'm trying to run rootless container with network in LXC container but it doesn't work.

~/.config/containers/systemd/CT.network

[Unit]
Description=CT network
After=network-online.target

[Network]
NetworkName=CT
Subnet=192.168.100.0/24
Gateway=192.168.100.1

[Install]
WantedBy=default.target

~/.config/containers/systemd/CT.container

[Unit]
Description=CT container
After=local-fs.target

[Container]
ContainerName=CT
Image=ct:latest
Network=CT.network
IP=192.168.100.10
HostName=CT
UserNS=keep-id:uid=1000,gid=1000

[Install]
WantedBy=multi-user.target default.target

$ podman network ls
NETWORK ID    NAME        DRIVER
c670f77283e5  CT          bridge
2f259bab93aa  podman      bridge

$ podman network inspect CT
[
     {
          "name": "CT",
          "id": "c670f77283e5d64b4937bb1ac6f2757e025295f0ea5c46d06261983cd5f19e48",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-10-15T13:48:12.357029846Z",
          "subnets": [
               {
                    "subnet": "192.168.100.0/24",
                    "gateway": "192.168.100.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

$ podman run --name=CT --replace --rm --network=CT --userns keep-id:uid=1000,gid=1000 --ip 192.168.100.10 ct:latest
ERRO[0000] IPAM error: could not find network "CT"
ERRO[0000] Unable to clean up network for container 4026db9d90b66deb0523fb28da627d79f27744e5e07469481108a6b26a26bcbb: "tearing down network namespace configuration for container 4026db9d90b66deb0523fb28da627d79f27744e5e07469481108a6b26a26bcbb: failed to convert net opts: unable to find network with name or ID CT: network not found"
Error: unable to find network with name or ID CT: network not found

Note that containers without Network work fine.

TLDR: ERRO[0000] IPAM error: could not find network "CT"

[Luap99]

Please provide the output of podman info

[davispuh]

host:
  arch: amd64
  buildahVersion: 1.33.8
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-150500.9.9.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 87.96
    systemPercent: 4.47
    userPercent: 7.57
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-leap
    version: "15.6"
  eventLogger: file
  freeLocks: 2038
  hostname: Aviola
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 475
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 490
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.12-2-pve
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 369164288
  memTotal: 6442450944
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-150500.3.6.1.x86_64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.11.0
    package: netavark-1.11.0-150500.3.6.1.x86_64
    path: /usr/lib/podman/netavark
    version: netavark 1.11.0
  ociRuntime:
    name: runc
    package: runc-1.1.14-150000.70.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.14
      commit: v1.1.14-0-g2c9f5602f0ba
      spec: 1.0.2-dev
      go: go1.22.6
      libseccomp: 2.5.3
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/490/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-150600.1.5.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.3
  swapFree: 536870912
  swapTotal: 536870912
  uptime: 0h 4m 33.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /var/lib/ct/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/ct/.local/share/containers/storage
  graphRootAllocated: 31526391808
  graphRootUsed: 12455993344
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-490/containers
  transientStore: false
  volumePath: /var/lib/ct/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.5
  Built: 1727956800
  BuiltTime: Thu Oct  3 12:00:00 2024
  GitCommit: ""
  GoVersion: go1.23.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.5

[Luap99]

see #22227 and #18162

With your error most likely the issue is the missing /var/lib/cni directory.

[davispuh]

Not sure if it somehow got created but it does exist and now it started working after I updated LXC config see my answer below.

[davispuh]

I added LXC config

lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.cgroup2.devices.allow: c 10:200 rwm

and now it started working. But weird thing is that afterwards I removed lxc.cgroup2.devices.allow: c 10:200 rwm but it still works.