Allowing k3s-agent to start containerd, when there is no access to k3s-server

[milan-mkip]

Hi,

I'm prototyping a system, where devices (k3s nodes) are in locations with inconsistent network access. I'm deploying pods to said nodes, and they remain running even during such connection outages.

However if the node happens to reboot during such connection outage, k3s-agent doesn't seem to run containerd and attempt to start the pods until the connection is recovered.

with journalctl -eu k3s-agent I could see the following error repeat until the connection was made:

failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF

127.0.0.1:6444/cacerts seems to redirect to the api server which is not available until there is a network connection...

After the connection is made, containerd starts and the node starts working as seen below.

Is there a way to configure k3s-agent to automatically run containerd and the previously known pods even after a reboot without network access?

Nov 07 16:10:12 node_1 k3s[765]: time="2024-11-07T16:10:12+02:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Nov 07 16:10:12 node_1 k3s[765]: time="2024-11-07T16:10:12+02:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: /var/lib/rancher/k3s/agent/serving-kubelet.crt: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Using private registry config file at /etc/rancher/k3s/registries.yaml"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Module overlay was already loaded"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Module br_netfilter was already loaded"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Using containerd template at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Nov 07 16:10:23 node_1 k3s[765]: time="2024-11-07T16:10:23+02:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Nov 07 16:10:24 node_1 k3s[765]: time="2024-11-07T16:10:24+02:00" level=info msg="containerd is now running"```

[brandond]

A connection to at least one server is required for the agent to start any of the embedded components. The agent receives configuration from the server, and this configuration is necessary to start both containerd and kubelet. We have considered in the past caching some configuration between runs, but this would create the possibility of the agent starting up with stale configuration that is out of sync with the rest of the cluster.

[milan-mkip]

Seems like this is common behavior in most kubernetes distributions. It's just a bit inconsistent in IoT/Edge use cases, as pods created from eg. a daemonset remain running on a node even if the node is isolated from the rest of the cluster. So sometimes the node can work with an out of sync configuration and sometimes it cannot.

Would it be possible to change the address that the worker node is using to query the configuration from the server? I'm just wondering if it would be possible for me to create a shim that forwards requests to the server, but in these cases, it could return the previously known configs, if the server was unavailable at startup.

[brandond]

I can't think of any reason why that wouldn't be possible with a moderate amount of code.

This particular use case isn't supported well by Kubernetes in general so it's not something we've prioritized addressing. The kubelet isn't really designed to be used in a partially-attached scenario, and other components like the controller-manager expect it to regularly update the node status to reflect that is it available.

If you only need to have a functional container runtime even when the node is offline, it might be easier to run a standalone instance of containerd, and just point k3s at that.

[milan-mkip]

With the standalone containerd and k3s pointed to that with the --container-runtime-endpoint the containers are visible after the restart.

Running those containers manually is a bit troublesome even with host network mode, sinche ctr -n k8s.io containers info <id>, shows that k3s/kubelet has created some dependencies to folders that don't exist while k3s isn't running.

            "namespaces": [

                {
                    "type": "ipc",
                    "path": "/proc/<pid>/ns/ipc"
                },
                {
                    "type": "uts",
                    "path": "/proc/<pid>/ns/uts"
                },
                {
                    "type": "network",
                    "path": "/proc/<pid>/ns/net"
                },
            ]

Those appear even when hostNetwork, hostIPC and hostPID are set to true, but I might not understand enough about kubernetes just yet to know if those are even relevant.

For a crude solution I could infer the general content of the containers from the info dump and recreate "vanilla" versions them without the kubernetes dependencies. Then when k3s has a connection, I could remove those containers after k3s has created the "kubernetes" versions of them.