[Release-1.30] - Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt

[dereknola]

Backport fix for Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt

• Secrets Reencrypt command throws fatal error client timing out when there are 1000 basic secrets to reencrypt #11384

[aganesh-suse]

Validated on release-1.30 branch with version v1.30.8-rc1+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 etcd, 2 cp and 1 agent node.

Describe the bug:

Etcd Config.yaml:

token: xxxx
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
node-taint:
- node-role.kubernetes.io/etcd:NoExecute
cluster-init: true
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
debug: true

Control Plane Config.yaml:

token: xxxx
server: https://1.1.1.1:6443
disable-etcd: true
node-taint:
- node-role.kubernetes.io/control-plane:NoSchedule
write-kubeconfig-mode: "0644"
secrets-encryption: true
node-external-ip: 4.4.4.4
node-label:
- k3s-upgrade=server
debug: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.30.8-rc1+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Create 1000 basic secrets:

echo 'this is a file' > file.txt && for i in {1..1000}; do echo test$i >> file.txt; kubectl create secret generic test$i --from-file=file.txt; done

5. Perform the secrets encryption operations: prepare/rotate/reencrypt (Reboot ALL nodes after every command - primary etcd first, all other etcd next, then all cp nodes)

$ sudo k3s secrets-encrypt prepare
$ sudo k3s secrets-encrypt rotate
$ sudo k3s secrets-encrypt reencrypt

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.30.8-rc1+k3s1 (b43a365f)
go version go1.22.9

There were no fatal errors found on running the prepare/rotate/reencrypt workflow.
Current Output of reencrypt command (that had failed previously):

$ sudo /usr/local/bin/k3s secrets-encrypt reencrypt
time="2024-12-13T04:21:23Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/c88d88febb62fa475e3ca28a2ddb6c98856c37f75d98a6998124835561e9c104"
time="2024-12-13T04:21:23Z" level=debug msg="Running /var/lib/rancher/k3s/data/c88d88febb62fa475e3ca28a2ddb6c98856c37f75d98a6998124835561e9c104/bin/k3s-secrets-encrypt [/usr/local/bin/k3s secrets-encrypt reencrypt]"
reencryption started

$ journalctl -xeu k3s | grep 'SecretsProgress' 
Dec 13 04:21:24 ip-172-31-7-5 k3s[9688]: I1213 04:21:24.359752    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 50 secrets"
Dec 13 04:21:24 ip-172-31-7-5 k3s[9688]: I1213 04:21:24.867312    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 100 secrets"
Dec 13 04:21:25 ip-172-31-7-5 k3s[9688]: I1213 04:21:25.535330    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 150 secrets"
Dec 13 04:21:26 ip-172-31-7-5 k3s[9688]: I1213 04:21:26.145012    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 200 secrets"
Dec 13 04:21:26 ip-172-31-7-5 k3s[9688]: I1213 04:21:26.768278    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 250 secrets"
Dec 13 04:21:27 ip-172-31-7-5 k3s[9688]: I1213 04:21:27.357821    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 300 secrets"
Dec 13 04:21:28 ip-172-31-7-5 k3s[9688]: I1213 04:21:28.009209    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 350 secrets"
Dec 13 04:21:28 ip-172-31-7-5 k3s[9688]: I1213 04:21:28.762586    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 400 secrets"
Dec 13 04:21:29 ip-172-31-7-5 k3s[9688]: I1213 04:21:29.415143    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 450 secrets"
Dec 13 04:21:30 ip-172-31-7-5 k3s[9688]: I1213 04:21:30.035259    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 500 secrets"
Dec 13 04:21:30 ip-172-31-7-5 k3s[9688]: I1213 04:21:30.696350    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 550 secrets"
Dec 13 04:21:31 ip-172-31-7-5 k3s[9688]: I1213 04:21:31.293043    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 600 secrets"
Dec 13 04:21:31 ip-172-31-7-5 k3s[9688]: I1213 04:21:31.985165    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 650 secrets"
Dec 13 04:21:32 ip-172-31-7-5 k3s[9688]: I1213 04:21:32.758132    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 700 secrets"
Dec 13 04:21:33 ip-172-31-7-5 k3s[9688]: I1213 04:21:33.477914    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 750 secrets"
Dec 13 04:21:34 ip-172-31-7-5 k3s[9688]: I1213 04:21:34.347606    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 800 secrets"
Dec 13 04:21:34 ip-172-31-7-5 k3s[9688]: I1213 04:21:34.968690    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 850 secrets"
Dec 13 04:21:35 ip-172-31-7-5 k3s[9688]: I1213 04:21:35.675357    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 900 secrets"
Dec 13 04:21:36 ip-172-31-7-5 k3s[9688]: I1213 04:21:36.469512    9688 event.go:389] "Event occurred" object="ip-172-31-7-5" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="SecretsProgress" message="reencrypted 950 secrets"

Post server restart status:

$ sudo /usr/local/bin/k3s secrets-encrypt status
time="2024-12-13T04:32:58Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/c88d88febb62fa475e3ca28a2ddb6c98856c37f75d98a6998124835561e9c104"
time="2024-12-13T04:32:58Z" level=debug msg="Running /var/lib/rancher/k3s/data/c88d88febb62fa475e3ca28a2ddb6c98856c37f75d98a6998124835561e9c104/bin/k3s-secrets-encrypt [/usr/local/bin/k3s secrets-encrypt status]"
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2024-12-13T04:06:18Z