Cannot get rid of disable-cloud-controller: true in k3s config

[mquhuy]

Hello, still me, the guy in #87 who is trying to make CABP3 work with Metal3.

I'm bumping into a situation in which no matter how I tried, I seem cannot get rid of the disable-cloud-controller: true in the /etc/rancher/k3s/config.yaml. This error makes the node tainted with label node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule and can't be scheduled into. I found this thread in k3s community which seems to describe my issue perfectly. (I'm pretty sure that is the main issue, as I could remove the taint by log in to the node, remove that line manually and restart k3s service)

I found that in https://github.com/cluster-api-provider-k3s/cluster-api-k3s/blob/main/pkg/k3s/config.go#L45, this field seems to be negative of serverConfig.disableExternalCloudProvider, so in my template, I specifically set it to true:

serverConfig:
    disableExternalCloudProvider: true

(This field is, btw, missing from https://github.com/cluster-api-provider-k3s/cluster-api-k3s/blob/main/samples/deployment/control-plane-k3s/v1.2.2/control-plane-components.yaml and https://github.com/cluster-api-provider-k3s/cluster-api-k3s/blob/main/samples/deployment/bootstrap-k3s/v1.2.2/bootstrap-components.yaml, so I did have to edit those files and put the local changes to clusterctl config).

However, despite this new field was accepted and displayed correctly when I describe my kthreescontrolplane object, the disable-cloud-controller: true still managed to end up in the node's k3s config. Could you please help me with this? Maybe there's something I should have/have not done? Thank you a lot in advance!

[mogliang]

disableExternalCloudProvider: true is the right setting. Below is our cr to enable builtin cloudprovider.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KThreesControlPlane
metadata:
  name: ${CLUSTER_NAME}-control-plane
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AksInfraMachineTemplate
    name: ${CLUSTER_NAME}-control-plane
  kthreesConfigSpec:
    serverConfig:
      disableExternalCloudProvider: true
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: ${KUBERNETES_VERSION}

can you verify your k3s controlplane yaml ?

[mquhuy]

Hi @mogliang . Thank you, my k3s controlplane yaml looks like this (it's a bit long, but basically disableExternalCloudProvider: true is there)

apiVersion: controlplane.cluster.x-k8s.io/v1beta1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
kind: KThreesControlPlane                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
metadata:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  name: test1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  namespace: metal3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
spec:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  infrastructureTemplate:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
    kind: Metal3MachineTemplate                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
    name: test1-controlplane                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  kthreesConfigSpec:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    agentConfig:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
      kubeletArgs:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
      - cgroup-driver=systemd                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
      - container-runtime-endpoint=unix:///var/run/crio/crio.sock                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
      - feature-gates=AllAlpha=false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
      - node-labels=metal3.io/uuid={{ ds.meta_data.uuid }}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
      - provider-id=metal3://{{ ds.meta_data.providerid }}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
      - runtime-request-timeout=5m                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
      nodeName: '{{ ds.meta_data.name }}'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    files:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
    - content: |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        #!/bin/bash                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
        set -e                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
        url="$1"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        dst="$2"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        filename="$(basename $url)"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
        tmpfile="/tmp/$filename"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        curl -sSL -w "%{http_code}" "$url" | sed "s:/usr/bin:/usr/local/bin:g" > /tmp/"$filename"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
        http_status=$(cat "$tmpfile" | tail -n 1)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
        if [ "$http_status" != "200" ]; then                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
          echo "Error: unable to retrieve $filename file";                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
          exit 1;                                                                                                           
        else                                                                                   
          cat "$tmpfile"| sed '$d' > "$dst";                                                                                                     
        fi                                                                                                                                       
      owner: root:root                                                                                                                           
      path: /usr/local/bin/retrieve.configuration.files.sh                                                                                                                                                                                                                                                                                                                                                                                                                                                 
      permissions: "0755"                                                                                                                        
    - content: |                                                                                                                                                           
        ! Configuration File for keepalived                                                                                                      
                                                                                                                            
        script k8s_api_check {                                                                                                                                                                                                                                                                
            script "curl -sk https://127.0.0.1:6443/healthz"                                                                                     
            interval 5                                                                         
            timeout 5                                                                                                                                                      
            rise 3                                                                                                        
            fall 3                                                                             
        }                                                                                                                                        
                                                                                               
        vrrp_instance VI_1 {                                                                                                                                               
            state MASTER                                                             
            interface eth1                                                                                                                                                 
            virtual_router_id 1                                                                            
            priority 101                                                                                                                                                   
            advert_int 1                                                                                                    
            virtual_ipaddress {                                                                            
                192.168.111.249                                                                                             
            }                                                                                                                                                                
            track_script {                                                                                                                                                   
                k8s_api_check                                                                                                                                                
            }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
        }                                                                                                                                                                    
      path: /etc/keepalived/keepalived.conf                                                                                                                                  
    - content: |                                                                                                                                                             
        [connection]                                                                                     
        id=eth0                                                                      
        type=ethernet                                                                                                                                                        
        interface-name=eth0                                                                                                                      
        master=ironicendpoint                                                                                                                                                                                      
        slave-type=bridge                                                                                                                                                                                          
      owner: root:root                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
      path: /etc/NetworkManager/system-connections/eth0.nmconnection                                                      
      permissions: "0600"                                                                                                                                                                                                                                                                                                                              
    - content: |                                                                                                                                                                                                                                                                                                                                       
        [connection]                                                                                                                                                       
        id=ironicendpoint                                                                                                                                                  
        interface-name=ironicendpoint                                                                                                                                      
        type=bridge                                                                  
        autoconnect=yes                                                                                  
        autoconnect-priority=1                                                                                                                                             
        [bridge]                                                                                                                                                             
        interface-name=ironicendpoint                                                                                                                                      
        stp=false                                                                                                                     
        [ipv4]                                                                       
        address1={{ ds.meta_data.provisioningIP }}/{{ ds.meta_data.provisioningCIDR }}                                                                                     
        method=manual                                                                
        [ipv6]                                                                       
        addr-gen-mode=eui64                                                          
        method=ignore                                                                                                                 
      owner: root:root                                                               
      path: /etc/NetworkManager/system-connections/ironicendpoint.nmconnection                           
      permissions: "0600"                                                                                                                                                    
    - content: |                                                                     
        [kubernetes]                                                                                                                                                       
        name=Kubernetes                                                                        
        baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64                                                                                                                                     
        enabled=1                                                                                                                                                            
        gpgcheck=1                                                                                                                                                                                                                                                           
        repo_gpgcheck=0                                                 
        gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg                                                                                    
      owner: root:root                                             
      path: /etc/yum.repos.d/kubernetes.repo                                                                                
      permissions: "0644"                                                                                                                                                                                             
    - content: |                                                                                                            
        [registries.search]                                                                                                 
        registries = ['docker.io']                                                                                          
                                                                                                         
        [registries.insecure]                                           
        registries = ['192.168.111.1:5000']                                                                               
      path: /etc/containers/registries.conf         
    postK3sCommands:                                                                  
    - useradd -m metal3                                                 
    - echo 'metal3:password' | chpasswd                                                                                               
    - usermod -aG sudo metal3                                                                                             
    - echo 'metal3 ALL=(ALL) NOPASSWD:ALL' | EDITOR='tee -a' visudo                                                                                                                                                   
    - mkdir /home/metal3/.ssh                                                         
    - chown metal3:metal3 /home/metal3/.ssh                                                                                                                                                                                                                                  
    - chmod 700 /home/metal3/.ssh                                                                                                                                                                                                                                            
    - echo <sha-pubkey> > /home/metal3/.ssh/authorized_keys                                                                                         
    - chown metal3:metal3 /home/metal3/.ssh/authorized_keys                                                                                      
    - chmod 600 /home/metal3/.ssh/authorized_keys                                                                                                
    preK3sCommands:                                                                   
    - systemctl restart NetworkManager.service                     
    - nmcli connection load /etc/NetworkManager/system-connections/ironicendpoint.nmconnection                                        
    - nmcli connection up ironicendpoint            
    - nmcli connection load /etc/NetworkManager/system-connections/eth0.nmconnection                                                  
    - nmcli connection up eth0                                     
    - rm /etc/cni/net.d/*                                          
    - curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --flannel-backend                                              
      none                                 
    - systemctl enable --now keepalived                                               
    - sleep 30                                      
    serverConfig:                          
      disableExternalCloudProvider: true                           
  replicas: 1                                                                         
  version: v1.28.1                                            

[mquhuy]

I found the issue: it was because I copied the setup from https://github.com/cluster-api-provider-k3s/cluster-api-k3s/tree/main/samples/deployment, which was using v0.1.5. After upgrading to v0.1.9, the k3s config turned out to be what I expected.

Thank you very much for the prompt support anyway!

[anmazzotti]

Hello @mquhuy
I was actually planning to deprecate the DisableExternalCloudProvider flag in the upcoming v1beta2 API version.

Before this flag was implemented, the --disable-cloud-provider flag was always added to k3s bootstrap and the cloud-provider=external kubelet arg was also always hardcoded in.

What I did not know at the time is that without the cloud-provider=external flag, k3s is going to write the Node.spec.providerID to something like k3s://my-hostname

This breaks the CAPI workflow, as the Infrastructure Provider is expected to set node's ProviderID (and can't be updated once set).
Note that CAPD directly removes the taint when setting the node's ProviderID, pretty much acting as a cloud provider.

For this reason I think that the original (without flag) implementation was the correct one. So always include --disable-cloud-provider and always include the cloud-provider=external kubelet arg.
Does this make sense?
If so, then the DisableExternalCloudProvider should be deprecated and eventually removed.

However, I wonder if there is a use case where we do want a cloud provider and we also want to explicitly set the cloud provider name.
In that case I would still deprecate DisableExternalCloudProvider, but then add a DisableCloudProvider flag (to toggle the --disable-cloud-provider k3s arg), and also add a CloudProviderName string (to set the cloud-provider={name} or omit this kubelet arg if the string is empty).

In such case you should be able to either configure:

serverConfig:
   disableCloudProvider: true
   cloudProviderName: external

or

serverConfig:
   disableCloudProvider: false
   cloudProviderName: my-cloud-provider

[mquhuy]

Hi @anmazzotti .

Thank you for the info! I think what you said makes sense, for the usecase that I described in the OP, it just means a change in the config name.