How to run Dnsmasq in container on MacOS

[gaufde]

Hello,

I am trying to set up dnsmasq for local subdomains on macOS. Basically, I'd like to do what is described in this blog post, but run everything in Podman containers.

When I try to run

podman run --detach --name dnsmasq \
	   --publish 127.0.0.1:53:53/udp \
	   --publish 127.0.0.1:53:53/tcp \
	   --volume "$(pwd)"/dnsmasq.conf:/etc/dnsmasq.conf \
	   --cap-add=NET_ADMIN \
	   dockurr/dnsmasq

I get the error Error: cannot listen on the UDP port: listen udp4 :53: bind: address already in use.

I've done some research and it seems like there are a few possible reasons why this is a problem. One is that the VM might have an unnecessary process bound to port 53: #17690 (comment)

So, I ran

Podman machine ssh
root@localhost:~# systemctl disable systemd-resolved.service
root@localhost:~# systemctl stop systemd-resolved
root@localhost:~# exit

After that I can see:

~/PhpstormProjects/local-dev-env
podman machine ssh ss -tulpn
Netid State  Recv-Q Send-Q             Local Address:Port Peer Address:PortProcess                                 
udp   UNCONN 0      0                   192.168.55.3:53        0.0.0.0:*    users:(("aardvark-dns",pid=1852,fd=11))
udp   UNCONN 0      0                      127.0.0.1:323       0.0.0.0:*    users:(("chronyd",pid=1021,fd=5))      
udp   UNCONN 0      0      [fd52:2a5a:747e:3acd::10]:53           [::]:*    users:(("aardvark-dns",pid=1852,fd=13))
udp   UNCONN 0      0                          [::1]:323          [::]:*    users:(("chronyd",pid=1021,fd=6))      
tcp   LISTEN 0      128                      0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1056,fd=3))         
tcp   LISTEN 0      4096                     0.0.0.0:80        0.0.0.0:*    users:(("conmon",pid=1859,fd=5))       
tcp   LISTEN 0      4096                     0.0.0.0:443       0.0.0.0:*    users:(("conmon",pid=1859,fd=6))       
tcp   LISTEN 0      1024                192.168.55.3:53        0.0.0.0:*    users:(("aardvark-dns",pid=1852,fd=12))
tcp   LISTEN 0      1024   [fd52:2a5a:747e:3acd::10]:53           [::]:*    users:(("aardvark-dns",pid=1852,fd=14))
tcp   LISTEN 0      128                         [::]:22           [::]:*    users:(("sshd",pid=1056,fd=4))         

I've also found this work around #23128 (comment), so I tried:

podman run --detach --name dnsmasq \
	   -p 127.0.0.1:53:53/udp -p 10.0.0.37:53:53/udp \
	   --volume "$(pwd)"/dnsmasq.conf:/etc/dnsmasq.conf \
	   --cap-add=NET_ADMIN \
	   dockurr/dnsmasq

but that didn't have any success either.

Does anyone have any ideas how I can get this working?

I'm running Podman 5.2.2 on MacOS 13.6.7. Also, Podman machine is running in rootful mode if that makes any difference.

[rhatdan]

@Luap99 PTAL

[Luap99]

This is complicated, first as you already noticed systemd-resolved is running and needs to be disabled. (I think we may want to do this by default in podman machine)

The second issue is aardvark-dns using port 53 as well for the bridge interface. And while binding a specific host ip works around that on linux it will not work with podman machine. This is because in side the VM the host ip doesn't exists and cannot be used as all the traffic is proxied via gvproxy into the main interface of the VM. So in order to make port forwarding work from the VM into the container we discard the host ip and thus bind all inside the VM (295d87b)

Now in order to work around that you need to configure aardvark-dns to use a different port via dns_bind_port in containers.conf and then it also will need a new netavark with containers/netavark#1080 in the VM otherwise aardvark-dns will no longer get any traffic it becomes not functional. This is not a issue if you do not use the container name resolution.

[Luap99]

The netavark fix was just merged so we first need to cut a new release, likely will take some weeks. Then we need the new version inside the machine image otherwise aardvark-dns would not work. But as long as you do not need the container names then it should work

So, in order to get it working, all I would really have to take care of myself is remove the systemd-resolved process and use a different port number for dns_bind_port in containers.conf?

I would propose we do this by default for the machine image, containers/podman-machine-os#18

Ideally I could do all of this with a custom FCOS ignition file once this regression #23544 is fixed.

Yes that issue should be fixed regardless

[gaufde]

Thanks for opening that additional issue @Luap99! Of course I'll have to wait a few weeks for a new release for the latest changes but at least that will remove one hurdle.

It does sound like I should figure out how to make my own ignition file since the proposed changes in containers/podman-machine-os#18 might take some time to be decided.

Is there a place where I can get info on how to create an ignition file with the right units so that podman machine start works properly?

The docs mention (emphasis mine):

--ignition-path

Fully qualified path of the ignition file.

If an ignition file is provided, the file is copied into the user’s CONF_DIR and renamed. Additionally, no SSH keys are generated, nor are any system connections made. It is assumed that the user does these things manually or handled otherwise.

I've been looking more at issue #23544 to see if I can learn enough Go to contribute, but I'm having trouble getting podman machine start to work and I think that is because my custom ignition file doesn't have the ready.service unit.

For example, I can modify Podman by removing these lines, and adding if err != nil to line 199. That will allow me to produce a new machine with my custom ignition file, however, podman machine start just hangs at "Starting machine" indefinitely.

Do I basically need to convert these lines into something that I can add to my butane file? https://github.com/containers/podman/blob/70f31281d6a978bc0d8b6a9d464357989040d303/pkg/machine/ignition/ready.go#L27C1-L28C107

[gaufde]

@Luap99 I also tried following your suggestions to see if I could get dnsmasq running in a container manually.

podman machine init
podman machine set --rootful
podman machine start
podman pull dockurr/dnsmasq
podman machine ssh systemctl disable systemd-resolved.service
podman machine ssh systemctl stop systemd-resolved
podman machine ssh 'echo "dns_bind_port=530" > /etc/containers/containers.conf'
podman run --detach --name dnsmasq \
	   --publish 127.0.0.1:53:53/udp \
	   --publish 127.0.0.1:53:53/tcp \
	   --volume "$(pwd)"/dnsmasq.conf:/etc/dnsmasq.conf \
	   --cap-add=NET_ADMIN \
	   dockurr/dnsmasq

However, that last command fails with Error: Post "http://gateway.containers.internal/services/forwarder/expose": dial tcp: lookup gateway.containers.internal: Temporary failure in name resolution

Similarly, if I try to pull another image after making all these changes I get:

podman pull caddy
Resolving "caddy" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/library/caddy:latest...
Error: initializing source docker://caddy:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution

Are both of these errors something that will be fixed by containers/netavark#1080?

[Luap99]

No, that just sounds like you resolv.conf was not updated after stopping systemd-resolved. You may need to restart the VM first or try restarting NetworkManager

[gaufde]

Ah, Thanks @Luap99! Your continued help is much appreciated.

I now have a working sequence of steps:

podman machine init
podman machine set --rootful
podman machine start
podman machine ssh 'echo "dns_bind_port=530" > /etc/containers/containers.conf'
podman machine ssh 'cat > /etc/systemd/resolved.conf <<EOL
[Resolve]
DNSStubListener=no
EOL'
podman machine ssh 'systemctl restart systemd-resolved'
podman run --detach --name dnsmasq \
	   --publish 53:53/udp \
	   --publish 53:53/tcp \
	   --volume "$(pwd)"/dnsmasq.conf:/etc/dnsmasq.conf \
	   --cap-add=NET_ADMIN \
	   dockurr/dnsmasq