IPv6 failure with podman trying to set up access to KinD's internal image registry.

[jmazzitelli]

Fedora 41, Podman 5.3.1, KinD 0.20.0

Following step 4 here to setup KinD registry (using podman, not docker):

https://kind.sigs.k8s.io/docs/user/local-registry/#create-a-cluster-and-registry

Run podman network connect kind kind-registry and this error appears - question is, what is wrong and how to fix this?

Error: netavark: add ip addr to container veth: failed to add ipv6 address, is
ipv6 enabled in the kernel?: Netlink error: Permission denied (os error 13)

Strange thing is - if I run that command again, it tells me it is already connected:

Error: container 87791aefa7337b1eb7ace3db6baff6b17... is already connected
to network kind: network is already connected

Note: IPv6 is enabled, and SELinux is completely disabled:

$ sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0

$ getenforce
Disabled

When I run podman --log-level=debug network connect kind kind-registry, it finds the network, sets some sysctl values it wants set without issue, then that error (with no other indications of what the problem is):

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called connect.PersistentPreRunE(podman --log-level=debug network connect kind kind-registry) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/jmazzite/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/11033/containers    
DEBU[0000] Using static dir /home/jmazzite/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/11033/libpod/tmp     
DEBU[0000] Using volume path /home/jmazzite/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Successfully loaded network kind: &{kind 1a10b012b7863a42772ddfb58d1bcc0241f630595f423fcb9807eabe70022a67 bridge podman1 2024-12-17 13:22:19.894810654 -0500 EST [{{{fc00:f853:ccd:e793:: ffffffffffffffff0000000000000000}} fc00:f853:ccd:e793::1 <nil>} {{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>}] [] true false true [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 2 networks               
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/etc/resolv.conf" 
[DEBUG netavark::network::validation] Validating network namespace...
[DEBUG netavark::commands::setup] Setting up...
[INFO  netavark::firewall] Using nftables firewall driver
[DEBUG netavark::network::bridge] Setup network kind
[DEBUG netavark::network::bridge] Container interface name: eth1 with IP addresses [fc00:f853:ccd:e793::a/64, 10.89.0.10/24]
[DEBUG netavark::network::bridge] Bridge name: podman1 with IP addresses [fc00:f853:ccd:e793::1/64, 10.89.0.1/24]
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv6.conf.all.forwarding to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth1/autoconf to 0
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth1/accept_dad to 0
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth1/arp_notify to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv4/conf/eth1/rp_filter to 2
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/veth3/accept_dad to 0
Error: netavark (exit code 1): add ip addr to container veth: failed to add ipv6 address, is ipv6 enabled in the kernel?: Netlink error: Permission denied (os error 13)
DEBU[0000] Shutting down engines                        
INFO[0000] Received shutdown.Stop(), terminating!        PID=128035

Run that command again, and it says the network is already connected (??) and fails-fast:

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called connect.PersistentPreRunE(podman --log-level=debug network connect kind kind-registry) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/jmazzite/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/11033/containers    
DEBU[0000] Using static dir /home/jmazzite/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/11033/libpod/tmp     
DEBU[0000] Using volume path /home/jmazzite/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Successfully loaded network kind: &{kind 1a10b012b7863a42772ddfb58d1bcc0241f630595f423fcb9807eabe70022a67 bridge podman1 2024-12-17 13:22:19.894810654 -0500 EST [{{{fc00:f853:ccd:e793:: ffffffffffffffff0000000000000000}} fc00:f853:ccd:e793::1 <nil>} {{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>}] [] true false true [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 2 networks               
Error: container 9207e977db96738345c39b11de0f5c3b3ac00b4c9d811883a63a1944d0f7f73a is already connected to network kind: network is already connected
DEBU[0000] Shutting down engines                        
INFO[0000] Received shutdown.Stop(), terminating!        PID=128181

Other than this, KinD is started and I can access it fine via "kubectl get ns" for example.

This is how /proc is mounted:

$ podman exec -it kind-registry mount | grep '/proc'
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,size=0k,uid=11033,gid=11033,inode64)
devtmpfs on /proc/kcore type devtmpfs (ro,nosuid,size=4096k,nr_inodes=8045172,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (ro,nosuid,size=4096k,nr_inodes=8045172,mode=755,inode64)
devtmpfs on /proc/latency_stats type devtmpfs (ro,nosuid,size=4096k,nr_inodes=8045172,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,size=0k,uid=11033,gid=11033,inode64)
devtmpfs on /proc/timer_list type devtmpfs (ro,nosuid,size=4096k,nr_inodes=8045172,mode=755,inode64)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)

$ podman exec -it kind-registry ls -ld /proc
dr-xr-xr-x  640 nobody   nobody           0 Dec 19 13:37 /proc

[jmazzitelli]

Hm. I think I know what the problem is. My host doesn't have ipv6 disabled, but the container does.

The answer: make sure --sysctl=net.ipv6.conf.all.disable_ipv6=1 is not set on the podman container. Or if it is set, the value should be "0"otherwise ipv6 will be disabled.