K8s infrastructure, with Prometheus, Thanos, Alertmanager and Grafana.
Fork the repo and update the line url: ssh://[email protected]/tenstad/k8s-infra
in clusters/dev/flux-system/gotk-sync.yaml
to your username/orgname.
Install k3d
:
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
Install flux
:
curl -s https://fluxcd.io/install.sh | sudo bash
Install mc
:
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin
Create a new cluster using k3d
:
k3d cluster create --k3s-arg "--disable=traefik@server:0"
Create a secret containing the slack webhook url:
kubectl create namespace flux-system
kubectl create secret -n flux-system generic kube-prometheus-stack-alertmanager --from-literal=slack-apiurl=<slack api url>
Configure Flux CI to sync from your GitHub repo:
export GITHUB_TOKEN=<github token>
flux bootstrap github \
--owner=<github username / orgname> \
--repository=k8s-infra \
--branch=main \
--path=./clusters/dev
Get Cluster IP and update /etc/hosts:
ip=$(docker network inspect k3d-k3s-default | jq -r '.[0].Containers[] | select(.Name=="k3d-k3s-default-server-0") | .IPv4Address' | cut -d "/" -f1)
for host in prometheus.cluster.local grafana.cluster.local karma.cluster.local thanos.cluster.local metrics-editor.cluster.local alertmanager.cluster.local minio.cluster.local minio-api.cluster.local locust.cluster.local kibana.cluster.local; do
sudo sed -i "/${host}/d" /etc/hosts
echo "${ip} ${host}" | sudo tee -a /etc/hosts
done
Create in-cluster S3 bucket for Thanos metrics storage:
mc alias set minio-cluster-local http://minio-api.cluster.local minioadmin minioadmin
mc admin info minio-cluster-local
mc mb minio-cluster-local/thanos
mc admin user add minio-cluster-local thanos secret0123456789
mc admin policy set minio-cluster-local readwrite user=thanos
mc admin user info minio-cluster-local thanos
mc admin user svcacct add --access-key access0123456789 --secret-key secret0123456789 minio-cluster-local thanos
Wait for pods to start:
watch kubectl get pods -A
Explore the services and try to update config in GitHub and push/merge to main
to apply.
prometheus.cluster.local
grafana.cluster.local
(admin/prom-operator)
karma.cluster.local
thanos.cluster.local
metrics-editor.cluster.local
alertmanager.cluster.local
minio.cluster.local
(minioadmin/minioadmin)
locust.cluster.local
kibana.cluster.local
k3d cluster delete
Delete GitHub fork.