Skip to content

Latest commit

 

History

History
189 lines (132 loc) · 7.1 KB

README.md

File metadata and controls

189 lines (132 loc) · 7.1 KB
Raw

Cluster API Provider k3s

Cluster API Provider k3s provides the following Cluster API (CAPI) providers:

  • Cluster API Bootstrap Provider k3s (CABP3) is responsible for generating the instructions (and encoding them as cloud-init) to turn a Machine into a Kubernetes Node; this implementation brings up k3s clusters instead of full kubernetes clusters.
  • Cluster API ControlPlane Provider k3s (CACP3) is responsible for managing the lifecycle of control plane machines for k3s; this implementation brings up k3s clusters instead of full kubernetes clusters.

Getting Started

Warning: Project and documentation are in an early stage, there is an assumption that a user of this provider is already familiar with Cluster API. Please consider contributing.

Prerequisites

Check out the general Cluster API Quickstart page to see the prerequisites for Cluster API.

Three main pieces are:

  1. Management cluster. In the samples/azure/azure-setup.sh script, k3d is used, but feel free to use kind as well .
  2. clusterctl. Please check out Cluster API Quickstart for instructions.
  3. Infrastructure specific prerequisites:

In this getting started guide we'll be using Docker as the infrastructure provider (CAPD).

Create a management cluster

  1. Ensure kind is installed (instructions)
  2. Create a kind configuration to expose the local docker socket:
cat > kind-cluster-with-extramounts.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: capi-quickstart
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
EOF
  1. Run the following command to create the cluster:
kind create cluster --config kind-cluster-with-extramounts.yaml
  1. Check that you can connect to the cluster
kubectl get pods -A

Install the providers

  1. Create a file called clusterctl.yaml in the $HOME/.config/cluster-api directory with the following content:
# NOTE: the following will be changed to use 'latest' in the future
providers:
  - name: "k3s"
    url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.2.1/bootstrap-components.yaml"
    type: "BootstrapProvider"
  - name: "k3s"
    url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.2.1/control-plane-components.yaml"
    type: "ControlPlaneProvider"

This configuration tells clusterctl where to look for the provider manifests. You could run clusterctl config -h to check default clusterctl configuration file path.

  1. Install the providers:
clusterctl init --bootstrap k3s --control-plane k3s --infrastructure docker
  1. Wait for the pods to start

Create a workload cluster

There are a number of different cluster templates in the samples directory.

Note: there is an issue with CAPD, it would be better you could do this setup beforehand. Cluster API with Docker - "too many open files".

  1. Run the following command to generate your cluster definition:
export KIND_IMAGE_VERSION=v1.30.0
clusterctl generate cluster --from samples/docker/cluster-template-quickstart.yaml test1 --kubernetes-version v1.30.2+k3s2 --worker-machine-count 2 --control-plane-machine-count 1 > cluster.yaml

NOTE: the kubernetes version specified with the k3s suffix +k3s2.

  1. Check the contents of the generated cluster definition in cluster.yaml
  2. Ensure the definition is valid by doing a dry run:
kubectl apply -f cluster.yaml --dry-run=server
  1. When you are happy apply the definition:
kubectl apply -f cluster.yaml

Check the workload cluster

  • Check the state of the CAPI machines:
kubectl get machine
  • Get the kubeconfig for the cluster:
clusterctl get kubeconfig test1 > workload-kubeconfig.yaml

Note: if you are using Docker Desktop, you need to fix the kubeconfig by running:

# Point the kubeconfig to the exposed port of the load balancer, rather than the inaccessible container IP.
sed -i -e "s/server:.*/server: https:\/\/$(docker port test1-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g" ./workload-kubeconfig.yaml
  • Connect to the child cluster
kubectl --kubeconfig workload-kubeconfig.yaml get pods -A

Deleting the workload cluster

When deleting a cluster created via CAPI you must delete the top level Cluster resource. DO NOT delete using the original file.

For the quick start:

kubectl delete cluster test1

Additional Samples

Cluster API k3s has been tested on AWS, Azure, AzureStackHCI, Nutanix, OpenStack, Docker and Vsphere environments.

  • To try out the Azure flow, fork the repo and look at samples/azure/azure-setup.sh.
  • To try out the AWS flow, fork the repo and look at samples/aws/aws-setup.sh.
  • To try out the Nutanix flow, fork the repo and look at samples/nutanix/nutanix-setup.sh.
  • To try out the OpenStack flow, fork the repo and look at samples/openstack/setup.sh.
  • To try out the Vsphere flow, fork the repo and look at samples/vsphere-capv/setup.sh.

Developer Setup

You could also build and install CABP3 and CACP3 from src:

# Build image with `dev` tag
make BOOTSTRAP_IMG_TAG=dev docker-build-bootstrap
make CONTROLPLANE_IMG_TAG=dev docker-build-controlplane

# Push image to your registry
export REGISTRY="localhost:5001"  # Set this to your local/remote registry
docker tag ghcr.io/k3s-io/cluster-api-k3s/controlplane-controller:dev ${REGISTRY}/controlplane-controller:dev
docker tag ghcr.io/k3s-io/cluster-api-k3s/bootstrap-controller:dev ${REGISTRY}/bootstrap-controller:dev
docker push ${REGISTRY}/controlplane-controller:dev
docker push ${REGISTRY}/bootstrap-controller:dev

# Install CAPI k3s to management cluster
make install-controlplane # install CRDs
make install-bootstrap
make CONTROLPLANE_IMG=${REGISTRY}/controlplane-controller CONTROLPLANE_IMG_TAG=dev deploy-controlplane  # deploy the component
make BOOTSTRAP_IMG=${REGISTRY}/bootstrap-controller BOOTSTRAP_IMG_TAG=dev deploy-bootstrap

For easy development, please refer to tilt-setup.md.

Roadmap

  • Support for External Databases
  • Fix Token Logic
  • Clean up Control Plane Provider Code
  • Post an issue!