Network issue: As I assign 10.88.0.2 to my pod and then from the second container onwards they are assigned 10.88.0.random, of which the app instance then fails to connect to the others. #14565

Folaht · 2022-06-10T15:06:42Z

Folaht
Jun 10, 2022

Hey there!

[Short version]

I don't think the --ip option in podman pod create ... is working as it should.
The static ip address is only assigned to the first container, the others are randomly assigned from the subnet pool.

[Long version]

Like always, I'm still trying to run an app that needs to run several instances of itself, called nodes in order for it to work.
So I tried to run a couple of nodes inside a container with various results.
Lately I've gone from several nodes to a container, to one node per container and several containers in a pod.

Here's what I happens:

$ sudo podman ps -a
CONTAINER ID  IMAGE                                          COMMAND     CREATED        STATUS                     PORTS                                        NAMES
bff45986a7d8  localhost/podman-pause:4.1.0-1652106728                    6 minutes ago  Up 6 minutes ago           192.168.178.26:12000-12003->12000-12003/udp  09a6ba4e035f-infra
9884945ab6b4  ghcr.io/safenetwork-community/safenode:latest              6 minutes ago  Up 6 minutes ago           192.168.178.26:12000-12003->12000-12003/udp  root_node
bc22079024fa  ghcr.io/safenetwork-community/safenode:latest              5 minutes ago  Exited (0) 46 seconds ago                                               join_node_1
8a6e68dc19c4  ghcr.io/safenetwork-community/safenode:latest              5 minutes ago  Exited (0) 4 minutes ago                                                join_node_2

The root node is started first and the join nodes are to connect with the root node.
They however time out after a couple of minutes.

So I then try to investigate by using

sudo podman commit join_node_1 debug/joinnode-ipv4
sudo podman run -d --pod my_pod --name debug_node debug/joinnode-ipv4
sudo podman exec -it debug_node /bin/bash

To my astonishment, the node didn't timeout and then I exited and did the following:

$ sudo podman start join_node_1
$ sudo podman start join_node_2
$ sudo podman ps -a
CONTAINER ID  IMAGE                                          COMMAND     CREATED         STATUS                    PORTS                                        NAMES
bff45986a7d8  localhost/podman-pause:4.1.0-1652106728                    36 minutes ago  Up 36 minutes ago         192.168.178.26:12000-12003->12000-12003/udp  09a6ba4e035f-infra
9884945ab6b4  ghcr.io/safenetwork-community/safenode:latest              36 minutes ago  Up 36 minutes ago         192.168.178.26:12000-12003->12000-12003/udp  root_node
bc22079024fa  ghcr.io/safenetwork-community/safenode:latest              35 minutes ago  Up 7 minutes ago                                                       join_node_1
8a6e68dc19c4  ghcr.io/safenetwork-community/safenode:latest              35 minutes ago  Exited (0) 6 minutes ago                                               join_node_2
3344733b4836  localhost/debug/joinnode-ipv4:latest                       29 minutes ago  Up 29 minutes ago         192.168.178.26:12000-12003->12000-12003/udp  debug_node

What's going on here!?
Well I've noticed that there is one key difference between the join node and the debug node container.

Here are the debug_node's ip addresses:

bash-5.1$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b2:cf:83:38:81:0e brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.2/16 brd 10.88.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b0cf:83ff:fe38:810e/64 scope link
       valid_lft forever preferred_lft forever

And here are the join_node_1's ip addresses:

bash-5.1$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 5a:47:bc:14:86:85 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.12/16 brd 10.88.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5847:bcff:fe14:8685/64 scope link
       valid_lft forever preferred_lft forever

So that raises the question, how do I assign these IPs and what does podman do with it?

This is my run script:

$ ./podrun.sh
sudo podman pod create --name my_pod --add-host my_host_ipv4:10.88.0.2 --ip 10.88.0.2 --network podman --publish 192.168.178.26:12000-12003:12000-12003/udp --replace --userns auto -v comnet_sjefolaht_ipv4_vol:/home/admin/.safe/share
09a6ba4e035f8b59067de29cd694aa1380c0988fd81dbd93f4cd6863cb0682e5
sudo podman run --name root_node --pod my_pod --env NETWORK_NAME=network_ipv4 --env LOG_LEVEL=trace --env NUM_NODES=1 --env SKIP_AUTO_PORT_FORWARDING=true --env IDLE_TIMEOUT_MSEC=5500 --env KEEP_ALIVE_INTERVAL_MSEC=4000 --env CON_IP=10.88.0.2 --env CON_PORT=12000 --env PUB_IP=195.240.113.178 --env PUB_PORT=12000 --env FIRST=true -d ghcr.io/safenetwork-community/safenode:latest
9884945ab6b424fa6211704b6822e5252853e1b01c41fd7cb4e8671a91e87385
...
sudo podman run --name join_node_1 --pod my_pod --env NETWORK_NAME=my_network_ipv4 --env LOG_LEVEL=trace --env NUM_NODES=1 --env SKIP_AUTO_PORT_FORWARDING=true --env IDLE_TIMEOUT_MSEC=5500 --env KEEP_ALIVE_INTERVAL_MSEC=4000 --env CON_IP=ip_con --env CON_PORT=12001 --env PUB_IP=195.240.113.178 --env PUB_PORT=12001 --env FIRST=false -d ghcr.io/safenetwork-community/safenode:latest
bc22079024fa654d23c20087614380ba9b4b5851779fa47a2684502923fd0298
Error: container "join_node_1" does not exist
sudo podman run --name join_node_2 --pod my_pod --env NETWORK_NAME=my_network_ipv4 --env LOG_LEVEL=trace --env NUM_NODES=1 --env SKIP_AUTO_PORT_FORWARDING=true --env IDLE_TIMEOUT_MSEC=5500 --env KEEP_ALIVE_INTERVAL_MSEC=4000 --env CON_IP=ip_con --env CON_PORT=12002 --env PUB_IP=195.240.113.178 --env PUB_PORT=12002 --env FIRST=false -d ghcr.io/safenetwork-community/safenode:latest
8a6e68dc19c47076bd647bf6ca8fc458fcc9aa300c483b781b63efd39e6bc5ba
Error: container "join_node_2" does not exist

As you can see, I have assigned --ip 10.88.0.2 to my pod and then attached all containers to the pod.
Shouldn't then the ip addresses of the other nodes not also be 10.88.0.2?

And just to be sure all containers belong to the pod:

$ sudo podman pod ps
POD ID        NAME                STATUS      CREATED         INFRA ID      # OF CONTAINERS
09a6ba4e035f  my_pod  Degraded    59 minutes ago  bff45986a7d8  5

And here's my network information:

$ sudo podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2022-06-10T17:09:06.489123167+02:00",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

I can also not use --ip individually for each container next to the pod as I get this error when I do:

...
ERRO[0000] IPAM error: failed to get ips for container ID 05fce36509db2263b3347c1304116ed55214186894d2d652b236b9f109a33a58 on network podman
Error: IPAM error: requested ip address 10.88.0.2 is already allocated to container ID

Answered by Folaht

Jun 15, 2022

I found the answer.

There still was a --network parameter in the podman run command of the join nodes from before I started to use podman pod.
It was only deleted in the print command.

View full answer

mheon · 2022-06-13T18:04:33Z

mheon
Jun 13, 2022
Maintainer

Can you provide a podman inspect of one of the containers joined to the pod? I suspect they're not joining the pod network namespace but instead are configuring their own networks.

0 replies

Folaht · 2022-06-13T23:43:32Z

Folaht
Jun 13, 2022
Author

I have a podman inspect of two containers, root_node and join_node_1.

Actual podname is pod_sjefolaht_ipv4

https://gist.github.com/Folaht/ac447058f08a6a89a2b79029d085230d

[update]

Here are all the differences I found:

root_node

          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "10.88.0.1",
               "IPAddress": "10.88.0.2",
               "IPPrefixLen": 16,
               ...
               "MacAddress": "5a:95:cd:fb:da:3c",
              "12000/udp": [
                         {
                              "HostIp": "192.168.178.26",
                              "HostPort": "12000"
                         }
                    ],
                    "12001/udp": [
                         {
                              "HostIp": "192.168.178.26",
                              "HostPort": "12001"
                         }
                    ],
                    "12002/udp": [
                         {
                              "HostIp": "192.168.178.26",
                              "HostPort": "12002"
                         }
                    ],
                    "12003/udp": [
                         {
                              "HostIp": "192.168.178.26",
                              "HostPort": "12003"
                         }
                    ]

join_node_1

          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
              ...
               "MacAddress": "",
               ...
               "Ports": {
                    "12000/tcp": null,
                    "443/tcp": null
               },

root_node

               "SandboxKey": "/run/netns/netns-241f8e97-dd33-1097-088e-ed12068465aa",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "10.88.0.1",
                         "IPAddress": "10.88.0.2",
                         "IPPrefixLen": 16,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "5a:95:cd:fb:da:3c",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "7742e838e771"
                         ]
                    }
               }

join_node_1

               "SandboxKey": "",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "",
                         "IPAddress": "",
                         "IPPrefixLen": 0,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "a8e83b16f404"
                         ]
                    }
               }

0 replies

Folaht · 2022-06-15T06:02:12Z

Folaht
Jun 15, 2022
Author

I found the answer.

There still was a --network parameter in the podman run command of the join nodes from before I started to use podman pod.
It was only deleted in the print command.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network issue: As I assign 10.88.0.2 to my pod and then from the second container onwards they are assigned 10.88.0.random, of which the app instance then fails to connect to the others. #14565

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Network issue: As I assign 10.88.0.2 to my pod and then from the second container onwards they are assigned 10.88.0.random, of which the app instance then fails to connect to the others. #14565

Folaht Jun 10, 2022

Replies: 3 comments

mheon Jun 13, 2022 Maintainer

Folaht Jun 13, 2022 Author

Folaht Jun 15, 2022 Author

Folaht
Jun 10, 2022

mheon
Jun 13, 2022
Maintainer

Folaht
Jun 13, 2022
Author

Folaht
Jun 15, 2022
Author