This is home to my personal Kubernetes lab cluster. Flux watches this Git repository and makes the changes to my cluster based on the manifests in the kubernetes directory. Renovate also watches this Git repository and creates pull requests when it finds updates to Docker images, Helm charts, and other dependencies.
There is a template over at onedr0p/flux-cluster-template if you wanted to try and follow along with some of the practices I use here.
My cluster is talos running on proxmox VMs. This is a semi hyper-converged cluster, workloads are sharing the same available resources on my nodes while I have a separate server for data storage.
🔸 Click here to see my Ansible playbooks and roles.
- cilium: Internal Kubernetes networking plugin
- cert-manager: Creates SSL certificates for services in my Kubernetes cluster
- external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider
- external-secrets: Managed Kubernetes secrets using 1Password Connect
- ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS
- sops: Managed secrets for Kubernetes, Ansible and Terraform which are commited to Git
- Democratic CSI: Provides block and NFS storage provisioning
- volsync: Backup and recovery of persistent volume claims
Flux watches my kubernetes folder (see Directories below) and makes the changes to my cluster based on the YAML manifests.
Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
This Git repository contains the following directories under kubernetes.
📁 kubernetes # Kubernetes cluster defined as code
├── 📁 talos # main cluster
│ ├─📁 apps # applications
│ ├─📁 bootstrap # bootstrap procedures
│ └─📁 flux # core flux configuration
└── 📁 truenas # Truenas k3s
├─📁 apps # applications
├─📁 bootstrap # bootstrap procedures
└─📁 flux # core flux configuration| Name | CIDR |
|---|---|
| Network VLAN | 10.1.1.0/24 |
| Servers VLAN | 10.1.2.0/24 |
| Talos external services (BGP) | 10.84.2.0/24 |
| Kubernetes pods | 172.16.0.0/16 |
| Kubernetes services | 10.100.0.0/16 |
While most of my infrastructure and workloads are selfhosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.
The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, ntfy, and Gatus. However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in and only saves me roughly $18/month.
| Service | Use | Cost |
|---|---|---|
| 1Password | Secrets with External Secrets | 73Eur/yr |
| Cloudflare | Domain, DNS and proxy management | Free |
| Fastmail | Email hosting | $75/yr |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Total: $12.75/mo |
In my cluster there are two ExternalDNS instances deployed. One is deployed with the ExternalDNS webhook provider for UniFi which syncs DNS records to my UniFi router. The other ExternalDNS instance syncs DNS records to Cloudflare only when the ingresses and services have an ingress class name of external and contain an ingress annotation external-dns.alpha.kubernetes.io/target. All local clients on my network use my UniFi router as the upstream DNS server.
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
|---|---|---|---|---|---|---|
| Unifi Cloud Gateway | 1 | - | - | - | Router | |
| Unifi USW-Enterprise-24-POE | 1 | - | - | - | - | Network Switch |
| Dell Optiplex 7040 | 4 | 512GB NVMe | - | 64GB | Debian 12 (PVE) | Virtualization Host |
| Cyberpower OR600ERM1U | 1 | - | - | - | - | UPS |
| QNAP TVS-682 | 1 | 2x256GB SATA | 2x512GB SSD + 4x4TB HDD | 32GB | TrueNAS Scale | NAS |
| ESP32+Ebyte 72 POE adapter | 1 | - | - | - | ESPHome | Zigbee adapter |
Thanks to all the people who donate their time to the Home Operations Discord community. A lot of inspiration for my cluster comes from the people that have shared their clusters using the k8s-at-home GitHub topic. Be sure to check out the kubesearch.dev for ideas on how to deploy applications or get ideas on what you can deploy.
See the awful commit history
See LICENSE