ansible-builder fails with <urlopen error [Errno -2] Name or service not known>

[mdhowle]

I am using Gitea with Gitea (act) runners. I am using this image to run the workflow in rootless podman on RHEL9.4

My goal is to automate building an Ansible Execution Environment (EE) when a new version is tagged in my git repository. In this EE, I include several Ansible collections from the RedHat Automation Hub, a private Automation Hub, and Galaxy. Note that both RH Automation Hub and my private Automation Hub are both hosted on redhat.com.

However, when it attempts to pull collections only found from galaxy.ansible.com (specifically community.general), it fails with: Failed to download collection tar from 'galaxy' due to the following unforeseen error: <urlopen error [Errno -2] Name or service not known>. <urlopen error [Errno -2] Name or service not known>. Collections downloaded from redhat.com work fine.

If I sync community.general to my private Automation Hub instead and remove Galaxy from the server_list in ansible.cfg, it works.

If I run ansible-builder outside of a container, it works.

If I run curl https://galaxy.ansible.com/ within the job container, it works.

Why does DNS appear to fail for ansible.com only? What can I do to troubleshoot this issue further?

Thanks

ansible.cfg

[galaxy]
server_list=automation_hub, private_hub, galaxy

[galaxy_server.automation_hub]
url=https://console.redhat.com/api/automation-hub/content/published/
auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token

[galaxy_server.private_hub]
url=https://hub.[REDACTED].ansiblecloud.redhat.com/api/galaxy/content/community/

[galaxy_server.galaxy]
url=https://galaxy.ansible.com/

podman info

host:
  arch: amd64
  buildahVersion: 1.33.8
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 97.28
    systemPercent: 1.09
    userPercent: 1.62
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: file
  freeLocks: 2047
  hostname: XXX
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 362144
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 362144
      size: 65536
  kernel: 5.14.0-427.37.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 14217732096
  memTotal: 16421961728
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1003/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1003/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: unconfined
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 1h 8m 15.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/gitea-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitea-runner/.local/share/containers/storage
  graphRootAllocated: 34292629504
  graphRootUsed: 14714920960
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1003/containers
  transientStore: false
  volumePath: /home/gitea-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1723107101
  BuiltTime: Thu Aug  8 04:51:41 2024
  GitCommit: ""
  GoVersion: go1.21.11 (Red Hat 1.21.11-1.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

[mdhowle]

I made a few more discoveries.

The container is not able to resolve *.s3.dualstack.us-east-1.amazonaws.com or s3.dualstack.us-east-1.amazonaws.com. It can resolve s3.amazonaws.com however. Very strange...

[mdhowle]

IPv6 is the issue. I had to re-create the custom podman network with podman network create --ipv6 podman_gitea. Also, during testing, I moved galaxy to the first in the server_list of ansible.cfg; I moved it to the last again.

$ podman network inspect podman_gitea

[
     {
          "name": "podman_gitea",
          "id": "5b9d6099d27606ffe89568b4fb9b87fdb9e54663aa6ef2505a94d5fb18f95403",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-09-27T11:28:38.89404406-04:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               },
               {
                    "subnet": "fdad:979:8be4:5b61::/64",
                    "gateway": "fdad:979:8be4:5b61::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

cat ~/.config/containers/containers.conf

[containers]

[network]
default_network = "podman_gitea"