failed to start rootless container: "malloc" issues cause core dump of program in the container

[rdelangh]

Given my infrastructure:

• bare metal server, 768GB RAM, 2 x CPU of 28 cores each (thus 56 in total), running Ubuntu 24.04LTS, kernel "6.8.0-31-generic"
• see below for the "podman info" output
• I have edited the file "/usr/share/containers/containers.conf" based on some suggestions that I found on the web, related to my issues:

[containers]
default_sysctls = [
  "net.ipv4.ping_group_range=0 0",
]
[engine]
cgroup_manager = "cgroupfs"
[machine]
cpus=2
memory=2048

• I have a set of BIND config files residing in "/store/central/dns/primary"
• trying to have rootless containers running
• starting this container is ok, but it exits immediately:

$ podman run -p 10053:53 --name my-bind9-container-prim --detach -e TZ=MET -v /store/central/dns/primary/cfg:/etc/bind docker.io/ubuntu/bind9:latest
7b4edb46358b751b48a38c79d839b50494853b1afb04cfb37d4ee837554a5faa
$ podman container ps -a
CONTAINER ID  IMAGE                          COMMAND     CREATED        STATUS                      PORTS                  NAMES
7b4edb46358b  docker.io/ubuntu/bind9:latest              7 seconds ago  Exited (139) 6 seconds ago  0.0.0.0:10053->53/tcp  my-bind9-container-prim

• when I launch the container in interactive mode and try to launch the binary "/usr/sbin/named" from within the container, via "strace", I see that it spawns xxx threads, and encounters "malloc" failures:

$ podman run -it -p 10053:53 --name bind9-container-prim1 --detach -e TZ=MET -v /store/central/dns/primary/cfg:/etc/bind docker.io/ubuntu/bind9:latest /bin/bash
6879d9f1ac2d715e917d672d3878f44473990f7861c78005aba7687cafc2e569
$ podman container ps -a
CONTAINER ID  IMAGE                          COMMAND     CREATED        STATUS        PORTS                  NAMES
c1521c1190fd  docker.io/ubuntu/bind9:latest  /bin/bash   2 minutes ago  Up 2 minutes  0.0.0.0:10053->53/tcp  bind9-container-prim
$ podman exec -it c1 /bin/bash
root@c1521c1190fd:/# dpkg -l|grep strace
root@c1521c1190fd:/# apt-get update >/dev/null && apt-get -y install strace >/dev/null 2>&1
root@c1521c1190fd:/# strace -f /usr/sbin/named -f -4 -u bind 2>&1 | tee /tmp/strace.out
...
[pid   263] futex(0x570526ce1268, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid   263] mprotect(0x7efda00da000, 8192, PROT_READ|PROT_WRITE) = -1 ENOMEM (Cannot allocate memory)
[pid   263] mmap(0x7efb00000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7efaf8000000
[pid   263] mprotect(0x7efaf8000000, 143360, PROT_READ|PROT_WRITE) = -1 ENOMEM (Cannot allocate memory)
[pid   263] munmap(0x7efaf8000000, 67108864) = 0
[pid   263] mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
...             
[pid   266] +++ killed by SIGABRT (core dumped) +++
[pid   265] +++ killed by SIGABRT (core dumped) +++
[pid   264] +++ killed by SIGABRT (core dumped) +++
[pid   263] +++ killed by SIGABRT (core dumped) +++
+++ killed by SIGABRT (core dumped) +++

The lines with "core dumped" occur for 114 different PIDs...

=> anyone has a clue what's causing this issue ?

PS: output from "podman info":

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.91
    systemPercent: 0.02
    userPercent: 0.07
  cpus: 56
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: file
  freeLocks: 2011
  hostname: xyz
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 10053
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 10053
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.8.0-31-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 784813744128
  memTotal: 811339337728
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: containernetworking-plugins_1.1.1+ds1-3build1_amd64
    path: /usr/lib/cni
  ociRuntime:
    name: runc
    package: runc_1.1.12-0ubuntu3_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12-0ubuntu3
      spec: 1.0.2-dev
      go: go1.22.2
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /home/bind/.config/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 233h 37m 37.00s (Approximately 9.71 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - quay.io
  - docker.io
store:
  configFile: /home/bind/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/bind/.local/share/containers/storage
  graphRootAllocated: 105089261568
  graphRootUsed: 21847810048
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 18
  runRoot: /home/bind/.config/containers
  transientStore: false
  volumePath: /home/bind/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

[rdelangh]

Even when running as "root", the same error occurs. So this seems not a privileges issue. Podman seems still to be wanting to run this container on 56 CPUs, altough we configured "/usr/share/containers/containers.conf" to use only 2 CPUs:
...
[machine]
cpus=2
memory=2048

Is this file being ignored ?

[Luap99]

The [machine] section is for podman machine settings not containers.

[rdelangh]

@Luap99 : ok Paul, I will take these out;
alternatively, where/how would I have to tune something to fix the issue that I am encountering ?

[Luap99]

Sorry I have no idea about the mmap problem. If you want to limit the memory for this container you can try use --memory 2048MB on the podman run command

[rdelangh]

I tried this already before, but this seems to have an issue when running the container 'rootless':

podman run -p 10053:53 --cpus 2 --memory 2048MB --name bind9-container-prim --detach -e TZ=MET -v /store/central/dns/primary/cfg:/etc/bind docker.io/ubuntu/bind9:latest

produced this error:

Error: runc: runc create failed: unable to start container process: unable to apply cgroup configuration: rootless needs no limits + no cgrouppath when no permission is granted for cgroups: mkdir /sys/fs/cgroup/user.slice/user-0.slice/d93b3d2fb62cca60d4fa439e889935ee2d67058b9bcde4f96cbf9bb2ccc25ee9: permission denied: OCI permission denied

The container did not startup:

CONTAINER ID  IMAGE                          COMMAND     CREATED        STATUS      PORTS                  NAMES
d93b3d2fb62c  docker.io/ubuntu/bind9:latest              2 seconds ago  Created     0.0.0.0:10053->53/tcp  bind9-container-prim

[rdelangh]

I found a thread on the Redhat subjects, describing exactly same issue:
https://issues.redhat.com/browse/RHEL-5870

They found that this is an issue related to "systemd":
systemd/systemd#9512

which would be fixed in 2018 with this commit:
poettering/systemd@fed374a

I am encountering this issue on our fresh Ubuntu 24.04, which comes with "systemd" version 255.4

When I invoke the "podman start" command with a lot of verbosity, for that "Created" but not started container in my previous comment above, then I get the following errors:

% podman --log-level=debug start d9 2>&1
...
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container d9...
...