Connection reset by peer when connecting to published port

[rbjorklin]

Issue Description

Seeing Connection reset by peer when trying to connect to application inside container.

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

$curl http://127.0.0.1:8080/liveness
curl: (56) Recv failure: Connection reset by peer

$ ss -tlpn '( sport = :8080 )'
State             Recv-Q            Send-Q                       Local Address:Port                         Peer Address:Port            Process
LISTEN            0                 4096                             127.0.0.1:8080                              0.0.0.0:*                users:(("rootlessport",pid=48154,fd=11))

$ podman exec -ti 3171005acd7e /bin/ash -c 'wget -qO- http://127.0.0.1:8080/liveness'
OK

Describe the results you expected

I expected the published port to be routed to my application and return OK as it does inside the container.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 91.12
    systemPercent: 2.46
    userPercent: 6.43
  cpus: 3
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "38"
  eventLogger: journald
  freeLocks: 2039
  hostname: trader-ocaml-ash
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.4.9-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 693510144
  memTotal: 3990401024
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1h 6m 11.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/rbjorklin/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 6
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rbjorklin/.local/share/containers/storage
  graphRootAllocated: 80487096320
  graphRootUsed: 4336164864
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 10
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/rbjorklin/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1691705273
  BuiltTime: Thu Aug 10 22:07:53 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Hetzner VM using Fedora 38

$ rpm -q podman
podman-4.6.1-1.fc38.x86_64

Additional information

Freshly installed system.

[rbjorklin]

podman run --log-level=debug

Output:

podman run --log-level=debug --name=trader --replace --rm --cgroups=split --network=systemd-trader --sdnotify=conmon -d -v /home/rbjorklin/config.yaml:/config.yaml --publish 127.0.0.1:8080:8080 --env CONDUIT_TLS=openssl ghcr.io/rbjorklin/trader-ocaml:latest --listen-prometheus=8081 --config-file=config.yaml
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level=debug --name=trader --replace --rm --cgroups=split --network=systemd-trader --sdnotify=conmon -d -v /home/rbjorklin/config.yaml:/config.yaml --publish 127.0.0.1:8080:8080 --env CONDUIT_TLS=openssl ghcr.io/rbjorklin/trader-ocaml:latest --listen-prometheus=8081 --config-file=config.yaml)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/rbjorklin/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/containers-user-1000/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/rbjorklin/.local/share/containers/storage
DEBU[0000] Using run root /tmp/containers-user-1000/containers
DEBU[0000] Using static dir /home/rbjorklin/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp
DEBU[0000] Using volume path /home/rbjorklin/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 10
DEBU[0000] Successfully loaded network systemd-trader: &{systemd-trader 732cdb9f01868e996e88f113f0c7b93177dad9576892a56c9809bf69293c1e9a bridge podman1 2023-08-12 03:36:37.606935964 +0000 UTC [{{{192.168.100.0 ffffff00}} 192.168.100.1 <nil>}] [] false false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 2 networks
DEBU[0000] Adding port mapping from 8080 to 8080 length 1 protocol ""
DEBU[0000] Pulling image ghcr.io/rbjorklin/trader-ocaml:latest (policy: always)
DEBU[0000] Looking up image "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "ghcr.io/rbjorklin/trader-ocaml:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Found image "ghcr.io/rbjorklin/trader-ocaml:latest" as "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage
DEBU[0000] Found image "ghcr.io/rbjorklin/trader-ocaml:latest" as "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage ([overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267)
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate ghcr.io/rbjorklin/trader-ocaml:latest for ghcr.io/rbjorklin/trader-ocaml:latest
DEBU[0000] parsed reference into "[overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]ghcr.io/rbjorklin/trader-ocaml:latest"
Trying to pull ghcr.io/rbjorklin/trader-ocaml:latest...
DEBU[0000] Copying source image //ghcr.io/rbjorklin/trader-ocaml:latest to destination image [overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]ghcr.io/rbjorklin/trader-ocaml:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "ghcr.io/rbjorklin/trader-ocaml:latest"
DEBU[0000] No credentials matching ghcr.io/rbjorklin/trader-ocaml found in /run/user/1000/containers/auth.json
DEBU[0000] Found credentials for ghcr.io/rbjorklin/trader-ocaml in credential helper containers-auth.json in file /home/rbjorklin/.config/containers/auth.json
DEBU[0000]  No signature storage configuration found for ghcr.io/rbjorklin/trader-ocaml:latest, using built-in default file:///home/rbjorklin/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/ghcr.io
DEBU[0000] GET https://ghcr.io/v2/
DEBU[0000] Ping https://ghcr.io/v2/ status 401
DEBU[0000] GET https://ghcr.io/token?account=rbjorklin&scope=repository%3Arbjorklin%2Ftrader-ocaml%3Apull&service=ghcr.io
DEBU[0000] Increasing token expiration to: 60 seconds
DEBU[0000] GET https://ghcr.io/v2/rbjorklin/trader-ocaml/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0000] Using blob info cache at /home/rbjorklin/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image docker:ghcr.io/rbjorklin/trader-ocaml:latest
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
DEBU[0000] Downloading /v2/rbjorklin/trader-ocaml/blobs/sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] GET https://ghcr.io/v2/rbjorklin/trader-ocaml/blobs/sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
Getting image source signatures
DEBU[0000] Reading /home/rbjorklin/.local/share/containers/sigstore/rbjorklin/trader-ocaml@sha256=43de0f56e2566ff6c2969493dc5654d73489ac3786375710a389a03da4f9c1fb/signature-1
DEBU[0000] Not looking for sigstore attachments: disabled by configuration
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:bbebc24ced046ecf759572941f6e0eb148442ef2c160ed9e9c5c65d0c537fc9a: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Skipping blob sha256:bbebc24ced046ecf759572941f6e0eb148442ef2c160ed9e9c5c65d0c537fc9a (already present):
DEBU[0000] Checking if we can reuse blob sha256:267092e23e6078e8a233820d03e147f1ae100bb53cf20c6fe518a3788c312268: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Skipping blob sha256:267092e23e6078e8a233820d03e147f1ae100bb53cf20c6fe518a3788c312268 (already present):
Copying blob bbebc24ced04 skipped: already exists
DEBU[0000] Checking if we can reuse blob sha256:3ed8244c7700b5b5c52d06f58929729194a0f168667be56b21395e9ba0b30081: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
Copying blob bbebc24ced04 skipped: already exists
Copying blob 267092e23e60 skipped: already exists
Copying blob bbebc24ced04 skipped: already exists
Copying blob bbebc24ced04 skipped: already exists
Copying blob 267092e23e60 skipped: already exists
Copying blob 3ed8244c7700 skipped: already exists
Copying blob 31e352740f53 skipped: already exists
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
Copying config 1cd3fe3a7f done
Writing manifest to image destination
DEBU[0000] setting image creation date to 2023-08-11 21:18:32.663789576 -0700 -0700
DEBU[0000] reusing image ID "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] saved image metadata "{}"
DEBU[0000] added name "ghcr.io/rbjorklin/trader-ocaml:latest" to image "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Pulled candidate ghcr.io/rbjorklin/trader-ocaml:latest successfully
DEBU[0000] Looking up image "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" in local containers storage
DEBU[0000] Trying "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" ...
DEBU[0000] parsed reference into "[overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Found image "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" as "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" in local containers storage
DEBU[0000] Found image "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" as "1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267" in local containers storage ([overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267)
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] User mount /home/rbjorklin/config.yaml:/config.yaml options []
DEBU[0000] Looking up image "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "ghcr.io/rbjorklin/trader-ocaml:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Found image "ghcr.io/rbjorklin/trader-ocaml:latest" as "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage
DEBU[0000] Found image "ghcr.io/rbjorklin/trader-ocaml:latest" as "ghcr.io/rbjorklin/trader-ocaml:latest" in local containers storage ([overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267)
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Inspecting image 1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Inspecting image 1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] Inspecting image 1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] Inspecting image 1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] Inspecting image 1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267
DEBU[0000] using systemd mode: false
DEBU[0000] setting container name trader
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Adding mount /proc
DEBU[0000] Adding mount /dev
DEBU[0000] Adding mount /dev/pts
DEBU[0000] Adding mount /dev/mqueue
DEBU[0000] Adding mount /sys
DEBU[0000] Adding mount /sys/fs/cgroup
DEBU[0000] Allocated lock 7 for container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1
DEBU[0000] parsed reference into "[overlay@/home/rbjorklin/.local/share/containers/storage+/tmp/containers-user-1000/containers]@1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] exporting opaque data as blob "sha256:1cd3fe3a7fd9e8d0f520303576dcfe4f0015fd768051b408af7a3627cf93c267"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1"
DEBU[0000] Container "eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1" has work directory "/home/rbjorklin/.local/share/containers/storage/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata"
DEBU[0000] Container "eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1" has run directory "/tmp/containers-user-1000/containers/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata"
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=lowerdir=/home/rbjorklin/.local/share/containers/storage/overlay/l/55EMCKSQCTYOJGLUA77G7EB3SR:/home/rbjorklin/.local/share/containers/storage/overlay/l/GVO4XJPC7E5A7QGKPJIUV5B2QT:/home/rbjorklin/.local/share/containers/storage/overlay/l/CSPOENK7GCCMI52RBTPKXPFGMM:/home/rbjorklin/.local/share/containers/storage/overlay/l/SE7VJKFYYAETBEZ76DMVY2HZJQ,upperdir=/home/rbjorklin/.local/share/containers/storage/overlay/02d9e615b190230acb75984d7214a4a10d357a564192e4a364f3ee9ca9a602c4/diff,workdir=/home/rbjorklin/.local/share/containers/storage/overlay/02d9e615b190230acb75984d7214a4a10d357a564192e4a364f3ee9ca9a602c4/work,,userxattr,volatile,context="system_u:object_r:container_file_t:s0:c48,c626"
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-e7c156ae-e50a-7da1-2bf8-26d1c91e56e2 for container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/run/systemd/resolve/stub-resolv.conf"
DEBU[0000] Mounted container "eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1" at "/home/rbjorklin/.local/share/containers/storage/overlay/02d9e615b190230acb75984d7214a4a10d357a564192e4a364f3ee9ca9a602c4/merged"
DEBU[0000] Created root filesystem for container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 at /home/rbjorklin/.local/share/containers/storage/overlay/02d9e615b190230acb75984d7214a4a10d357a564192e4a364f3ee9ca9a602c4/merged
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[INFO  netavark::firewall] Using iptables firewall driver
[DEBUG netavark::network::bridge] Setup network systemd-trader
[DEBUG netavark::network::bridge] Container interface name: eth0 with IP addresses [192.168.100.26/24]
[DEBUG netavark::network::bridge] Bridge name: podman1 with IP addresses [192.168.100.1/24]
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth0/autoconf to 0
[INFO  netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 192.168.100.1, metric 100)
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-D70F703E915DD exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-D70F703E915DD exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_ISOLATION_2 exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_ISOLATION_2 exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_ISOLATION_3 exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_ISOLATION_3 exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::firewall::varktables::helpers] rule -d 192.168.100.0/24 -j ACCEPT exists on table nat and chain NETAVARK-D70F703E915DD
[DEBUG netavark::firewall::varktables::helpers] rule ! -d 224.0.0.0/4 -j MASQUERADE exists on table nat and chain NETAVARK-D70F703E915DD
[DEBUG netavark::firewall::varktables::helpers] rule -s 192.168.100.0/24 -j NETAVARK-D70F703E915DD exists on table nat and chain POSTROUTING
[DEBUG netavark::firewall::varktables::helpers] rule -d 192.168.100.0/24 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT exists on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::firewall::varktables::helpers] rule -s 192.168.100.0/24 -j ACCEPT exists on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.conf.podman1.route_localnet to 1
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-DN-D70F703E915DD exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-DN-D70F703E915DD exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[DEBUG netavark::firewall::varktables::helpers] rule -j MARK  --set-xmark 0x2000/0x2000 exists on table nat and chain NETAVARK-HOSTPORT-SETMARK
[DEBUG netavark::firewall::varktables::helpers] rule -j MASQUERADE -m comment --comment 'netavark portfw masq mark' -m mark --mark 0x2000/0x2000 exists on table nat and chain NETAVARK-HOSTPORT-MASQ
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 192.168.100.0/24 -p tcp --dport 8080 -d 127.0.0.1 created on table nat and chain NETAVARK-DN-D70F703E915DD
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 127.0.0.1 -p tcp --dport 8080 -d 127.0.0.1 created on table nat and chain NETAVARK-DN-D70F703E915DD
[DEBUG netavark::firewall::varktables::helpers] rule -j DNAT -p tcp --to-destination 192.168.100.26:8080 --destination-port 8080 -d 127.0.0.1 created on table nat and chain NETAVARK-DN-D70F703E915DD
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-DN-D70F703E915DD -p tcp --dport 8080 -m comment --comment 'dnat name: systemd-trader id: eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1' created on table nat and chain NETAVARK-HOSTPORT-DNAT
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-DNAT -m addrtype --dst-type LOCAL exists on table nat and chain PREROUTING
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-DNAT -m addrtype --dst-type LOCAL exists on table nat and chain OUTPUT
[DEBUG netavark::commands::setup] {
        "systemd-trader": StatusBlock {
            dns_search_domains: Some(
                [
                    "dns.podman",
                ],
            ),
            dns_server_ips: Some(
                [
                    192.168.100.1,
                ],
            ),
            interfaces: Some(
                {
                    "eth0": NetInterface {
                        mac_address: "02:5e:f3:26:8e:ff",
                        subnets: Some(
                            [
                                NetAddress {
                                    gateway: Some(
                                        192.168.100.1,
                                    ),
                                    ipnet: 192.168.100.26/24,
                                },
                            ],
                        ),
                    },
                },
            ),
        },
    }
[DEBUG netavark::commands::setup] "Setup complete"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg="Starting parent driver"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg="opaque=map[builtin.readypipepath:/tmp/podman-run-1000/libpod/tmp/rootlessport3327792562/.bp-ready.pipe builtin.socketpath:/tmp/podman-run-1000/libpod/tmp/rootlessport3327792562/.bp.sock]"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg="Starting child driver in child netns (\"/proc/self/exe\" [rootlessport-child])"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg="Waiting for initComplete"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg="initComplete is closed; parent and child established the communication channel"
time="2023-08-12T04:53:07Z" level=info msg="Exposing ports [{127.0.0.1 8080 8080 1 tcp}]"
DEBU[0000] rootlessport: time="2023-08-12T04:53:07Z" level=info msg=Ready
DEBU[0000] rootlessport is ready
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/home/rbjorklin/.local/share/containers/storage/overlay/02d9e615b190230acb75984d7214a4a10d357a564192e4a364f3ee9ca9a602c4/merged"
DEBU[0000] Created OCI spec for container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 at /home/rbjorklin/.local/share/containers/storage/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 -u eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 -r /usr/bin/crun -b /home/rbjorklin/.local/share/containers/storage/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata -p /tmp/containers-user-1000/containers/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata/pidfile -n trader --exit-dir /tmp/podman-run-1000/libpod/tmp/exits --full-attach -l journald --log-level debug --syslog --conmon-pidfile /tmp/containers-user-1000/containers/overlay-containers/eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/rbjorklin/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/containers-user-1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /tmp/podman-run-1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/rbjorklin/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: 51789
INFO[0000] Got Conmon PID as 51787
DEBU[0000] Created container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 in OCI runtime
DEBU[0000] Adding nameserver(s) from network status of '["192.168.100.1"]'
DEBU[0000] Adding search domain(s) from network status of '["dns.podman"]'
DEBU[0000] found local resolver, using "/run/systemd/resolve/resolv.conf" to get the nameservers
DEBU[0000] Starting container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1 with command [./trader_app.exe --listen-prometheus=8081 --config-file=config.yaml]
DEBU[0000] Started container eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1
DEBU[0000] Notify sent successfully
eb510f8ae18858a9d277bfa3373d1502f6ff4c2e26832181b16fd061721f39f1
DEBU[0000] Called run.PersistentPostRunE(podman run --log-level=debug --name=trader --replace --rm --cgroups=split --network=systemd-trader --sdnotify=conmon -d -v /home/rbjorklin/config.yaml:/config.yaml --publish 127.0.0.1:8080:8080 --env CONDUIT_TLS=openssl ghcr.io/rbjorklin/trader-ocaml:latest --listen-prometheus=8081 --config-file=config.yaml)
DEBU[0000] Shutting down engines

[rbjorklin]

By changing the bind address of the application inside the container from 127.0.0.1 to 0.0.0.0 it works. Is this expected? Maybe even a requirement?

[Luap99]

yes this is required

[seem8]

It seems to be fixed in Podman 5.2.2 (for example in Fedora 40), where Pasta is used for port mapping. Application can listen on localhost and Podman can map it to whatever with -p 0.0.0.0:8080:8080 .

Meanwhile I'm using ENV variable, which is changing application to listen everywhere and it fixes Podman 4.9.x port mapping.