Servicelb pooling and reverse proxying the k3s control plane

[vasdee]

I posted this question on SO a little while ago but thought I might ask here, since this seems to be a little more activity. I'm evaluating k3s to try and achieve network segmentation, in this case following ISA95, where a group of nodes will act as a 'sub cluster' of sorts. Each
'sub cluster' will basically run the same core components, but will only communicate to the network segment directly northwards from the current sub-cluster....... I promise there's a good reason behind all this though, but out of scope for this question.

There's two main components I need to get going for this, one of them is to host a mirrored registry on a 'sub cluster'. The other is to reverse proxy the control plane back to either the next higher 'sub cluster' or if already at the top of the stack, the server load balancer IP address.

So far I've managed to follow the docs to create a servicelb, that only runs on a subset of nodes, that represents 1 of the network segments. This allows me to host a container registry as a service that will provide container images for the nodes 'below'.

My next task is to try and get use the same servicelb setup, to reverse proxy the controlplane IP/CNAME. The problem is, unlike the registry where I can create a type: loadbalancer for the service and essentially target the specific nodes that are running the registry pods, type: ExternalName doesn't seem to want to play nice with that setup. This is where I'm getting stumped, on the surface it looks like ExternalName should do the job, but my reading has also lead me to EndpointSlices, which also look like they could do the same. Am I on the right track here, or would there be another alternate way to achieve what I'm trying to achieve?

Original question with a pretty diagram if that helps:
https://stackoverflow.com/questions/79222201/connecting-a-k3s-agent-via-another-agents-control-plane

[brandond]

I don't think I'm really getting what you're trying to do here. Are you trying to have agents connect to the apiserver THROUGH other agents somehow? To what end? Kubernetes aside, how are you handling the K3s supervisor API?

The CNI itself also assumes a fairly flat network where all nodes are reachable to each other, even if you jurry-rig a registry and apiserver somehow, I think there are a lot of other things that are still going to bypass what you've got set up here.

Kubernetes isn't really built to tier like this.

[vasdee]

Are you trying to have agents connect to the apiserver THROUGH other agents

Yes, that's exactly what I'm trying to achieve here, in order to isolate a subset of nodes to only operate with certain network segments and only communicate northwards to the higher network segment

The CNI itself also assumes a fairly flat network where all nodes are reachable to each other, even if you jurry-rig a registry and apiserver somehow, I think there are a lot of other things that are still going to bypass what you've got set up here.

This unfortunately is what I was ultimately hoping to prove/disprove as a final test in my proof-of-concept setup, once I got the control plane proxied. Nodes living within a cloud based VNET, but scattered between 3 or 4 isolated subnets with very limited firewalls between.

I was hoping based on what I had read about the hybrid/multi-cloud (https://docs.k3s.io/networking/distributed-multicloud) setups that this would be possible, without a lot of extra inter-node communications

[brandond]

No, that just uses tailscale as a VPN coordinator to build the wireguard mesh between nodes. The nodes still all end up having connectivity to each other.

Why are you trying to have agents talk through other agents, instead of just setting everything up to talk through tailscale and the servers?

[vasdee]

It's for an enterprise setup within a control systems network. ISA95 helps to isolate workloads and applications, in a nice cyber-friendly way. Also, the control systems network we are (maybe were is a more realistic term now) hoping to deploy K3S is completely air-gapped, so the cloud hosted 6443 servers and registry comms need to eventually be proxied via the level 3.5 (DMZ) level nodes. Lower level nodes in the level 3 and 2 layers, can only communicate to the layers above.

I have a better diagram of what I'm trying to achieve, which might explain it better, but I don't think I'd be allowed to post it publicly. I can send if you like