Please note that the competition status for CKS Challenges is ended. Please do not submit a solution. It will not be scored.
This is a two node kubernetes cluster. Using the kube-bench
utility, identify and fix all the issues that were reported as failed for the controlplane
and the worker node
components.
Inspect the issues in detail by clicking on the icons of the interactive architecture diagram on the right and complete the tasks to secure the cluster. Once done click on the Check
button to validate your work.
Click on each icon (in the lab) to see more details. Once done, click the Check button to test your work.
Do the tasks in this order:
-
kube-bench
- Download
kube-bench
from AquaSec and extract it under/opt
filesystem. Use the appropriate steps from the kube-bench docs to complete this task. - Run
kube-bench
with config directory set to/opt/cfg
and/opt/cfg/config.yaml
as the config file. Redirect the result to/var/www/html/index.html
file.
When this challenge was last updated created,
v0.9.2
of kube-bench was current, so we should download that version for best compatibility.Reveal
-
Download and place under
opt
curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.9.2/kube-bench_0.9.2_linux_amd64.tar.gz | tar -xz -C /opt
-
Run it
-
Create directory for report
mkdir -p /var/www/html
-
Execute with given configuration instructions
/opt/kube-bench --config-dir /opt/cfg --config /opt/cfg/config.yaml > /var/www/html/index.html
-
Despite the fact that we redirected the output to
index.html
, the file content is text and can be inspected like thisless /var/www/html/index.html
- Download
-
kubelet (node)
- Ensure that the permissions of the kubelet
config.yaml
file are set to 600 (node01)
Reveal
-
ssh
tonode01
ssh node01
-
Edit the kubelet configuration
chmod 600 /var/lib/kubelet/config.yaml
-
Return to
controlplane
nodeexit
- Ensure that the permissions of the kubelet
-
kube-controller-mananger
- Ensure that the
--profiling argument
is set to false
Reveal
-
Edit the manifest
vi /etc/kubernetes/manifests/kube-controller-manager.yaml
-
Add the following to the list of arguments in the
command
section of the pod spec:- --profiling=false
-
Save and exit from
vi
. Controller manager pod will restart in a minute or so
- Ensure that the
-
kube-scheduler
- Ensure that the
--profiling
argument is set to false
Do the exact same staps as above, but with
/etc/kubernetes/manifests/kube-scheduler.yaml
- Ensure that the
-
etcd
- Correct the
etcd
data directory ownership
Reveal
-
View the report as discussed in the
kube-bench
section above, and find the FAIL at section1.1.12
-
Verify the data directory by checking the
volumes
section of theetcd
pod static manifest for thehostPath
. -
Correct the ownership as directed
chown -R etcd:etcd /var/lib/etcd
- Correct the
-
kube-apiserver
- Ensure that the
--profiling
argument is set tofalse
- Ensure that the
--audit-log-path
argument is set to/var/log/apiserver/audit.log
- Ensure that the
--audit-log-maxage
argument is set to30
- Ensure that the
--audit-log-maxbackup
argument is set to10
- Ensure that the
--audit-log-maxsize
argument is set to100
Reveal
So this looks like a bunch of argument changes. Well it is, but there's a bit more work than that. If we tell the apiserver to open a log at a given directory, then that directory is expected to be on the host machine, i.e.
controlplane
itself. This means we also need to create avolume
andvolumeMount
to satisfy this criterion, and also the host directory must exist.-
The directory into which the log file will go needs to exist first
mkdir -p /var/log/apiserver
-
Edit the manifest file
vi /etc/kubernetes/manifests/kube-apiserver.yaml
-
Put in all the new arguments
- --profiling=false - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-path=/var/log/apiserver/audit.log - --audit-log-maxsize=100
-
Create a
volume
for the log file (add to existingvolumes
)volumes: - hostPath: path: /var/log/apiserver/audit.log type: FileOrCreate name: audit-log
-
Create a
volumeMount
for this volume (add to existingvolumeMounts
)volumeMounts: - mountPath: /var/log/apiserver/audit.log name: audit-log
-
Save and exit
vi
. Wait up to a minute for api server to restart. Be aware of how to debug a crashed apiserver if you muck it up!
- Ensure that the
As DevOps engineers, we love everything to be automated!
Automation Script
Paste this entire script to the lab terminal, sit back and enjoy!
When the script completes, you can press the Check
button and the lab will be complete!
{
start_time=$(date '+%s')
## kube-bench
# Install and run kube-bench
echo 'kube-bench'
curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.9.2/kube-bench_0.9.2_linux_amd64.tar.gz | tar -xz -C /opt
mkdir -p /var/www/html
echo "Running kube-bench"
/opt/kube-bench --config-dir /opt/cfg --config /opt/cfg/config.yaml > /var/www/html/index.html
## etcd
echo 'etcd'
chown -R etcd:etcd /var/lib/etcd
## kubelet
echo 'kubelet'
ssh node01 'chmod 600 /var/lib/kubelet/config.yaml'
## kube-controller-mananger
echo 'kube-controller-mananger'
yq -i e '.spec.containers[0].command += "--profiling=false"' /etc/kubernetes/manifests/kube-controller-manager.yaml
## kube-scheduler
echo 'kube-scheduler'
yq -i e '.spec.containers[0].command += "--profiling=false"' /etc/kubernetes/manifests/kube-scheduler.yaml
## kube-apiserver
echo 'kube-apiserver'
# Create audit log path
mkdir -p /var/log/apiserver
# Patch api-server
yq e '.spec.containers[0].command += [
"--profiling=false",
"--audit-log-maxage=30",
"--audit-log-maxbackup=10",
"--audit-log-path=/var/log/apiserver/audit.log",
"--audit-log-maxsize=100"
] |
.spec.volumes += {"name": "audit-log", "hostPath":{"path":"/var/log/apiserver/audit.log", "type":"FileOrCreate"}} |
.spec.containers[0].volumeMounts += {"mountPath": "/var/log/apiserver/audit.log", "name": "audit-log"}' \
/etc/kubernetes/manifests/kube-apiserver.yaml > \
kube-apiserver.yaml.out
# Save current API server container ID
api_container_id=$(crictl ps | grep apiserver | cut -f 1 -d ' ')
mv -f kube-apiserver.yaml.out /etc/kubernetes/manifests/kube-apiserver.yaml
# Kick kubelet - I have seen it not notice the manifest change here.
systemctl restart kubelet
# Wait for API server restart (gets a new container ID)
new_id=''
while [ -z "$new_id" -o "$api_container_id" = "$new_id" ]
do
sleep 2
new_id=$(crictl ps | grep apiserver | cut -f 1 -d ' ')
echo "API server container id is $new_id"
done
sleep 5
kubectl get pods -n kube-system
end_time=$(date '+%s')
duration=$(( end_time - start_time ))
echo "Complete in ${duration}s"
}