Deploying k3s onto 11 raspberry pi's

[TomTucka]

Expected Behavior

k3s should be installed correctly and come up in HA mode

Current Behavior

TASK [k3s/master : Verify that all nodes actually joined (check k3s-init.service if this fails)] ***********************************************************************************************************FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (18 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (18 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (17 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (17 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (16 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (16 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (15 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (15 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (14 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (14 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (13 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (13 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (12 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (12 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (11 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (11 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (10 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (10 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (9 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (9 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (8 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (8 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (7 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (7 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (6 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (6 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (5 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (5 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (4 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (4 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (3 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (3 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (2 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (2 retries left).
FAILED - RETRYING: [10.20.40.20]: Verify that all nodes actually joined (check k3s-init.service if this fails) (1 retries left).
FAILED - RETRYING: [10.20.40.21]: Verify that all nodes actually joined (check k3s-init.service if this fails) (1 retries left).
fatal: [10.20.40.20]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.289780", "end": "2023-06-23 18:08:40.213489", "msg": "non-zero return code", "rc": 1, "start": "2023-06-23 18:08:39.923709", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
fatal: [10.20.40.21]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.284601", "end": "2023-06-23 18:08:50.878894", "msg": "non-zero return code", "rc": 1, "start": "2023-06-23 18:08:50.594293", "stderr": "The connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}

Context (variables)

Operating system: Raspberry PI OS Lite 64-Bit

Hardware: 11x Raspberry Pi Model 4b's 4gb RAM

Variables Used

all.yml

---
k3s_version: v1.25.9+k3s1
# this is the user that has ssh access to these machines
ansible_user: pi
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "Your/Timezone"

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.20.40.40"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "fU7dphX6YiHVNqo@"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  --flannel-iface={{ flannel_iface }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.5.12"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "192.168.30.1"

# image tag for metal lb
metal_lb_frr_tag_version: "v7.5.1"
metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"

# metallb ip range for load balancer
metal_lb_ip_range: "10.20.40.40-10.20.40.50"

# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficent compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have partiularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
  - 200
  - 201
  - 202
  - 203
  - 204

Hosts

host.ini

[master]
10.20.40.20
10.20.40.21

[node]
10.20.40.22
10.20.40.23
10.20.40.24
10.20.40.25
10.20.40.26
10.20.40.27
10.20.40.28
10.20.40.29
10.20.40.30

[k3s_cluster:children]
master
node

Possible Solution

• I've checked the General Troubleshooting Guide

[SeanRiggs]

Your VIP IP seems to be in the same range as your metallb:

apiserver_endpoint: "10.20.40.40"

metal_lb_ip_range: "10.20.40.40-10.20.40.50"

Make you VIP IP outside of your MetalLP range and try to rerun.

Also, dont forget to set:

system_timezone: "Your/Timezone"