Why a service account key file in the kube-apiserver is a private key?

[mazzy89]

According to the documentation the service-account-key-file flag should point to a public key https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#token-controller

You must pass a service account private key file to the token controller in the kube-controller-manager using the --service-account-private-key-file flag. The private key is used to sign generated service account tokens. Similarly, you must pass the corresponding public key to the kube-apiserver using the --service-account-key-file flag. The public key will be used to verify the tokens during authentication.

However looking at the kube-apiserver flags

kube-apiserver
...
    --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key

❯ cat service.key
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: service.key
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ -----BEGIN RSA PRIVATE KEY-----

it points to a private key. How come this is even possible that work? I would expect that the CM points to a private key while the API Server to a public key.

[brandond]

Honestly I'm not sure; it's been that way since the project was started:
https://github.com/k3s-io/k3s/blob/184f69732f08ebba1bdff69d4c7fa54bdea28dac/pkg/daemons/control/server.go

Most of this behavior was taken from upstream's hack/local-up-cluster.sh, which you can see currently uses only a single private key for both flags:
https://github.com/kubernetes/kubernetes/blob/release-1.22/hack/local-up-cluster.sh#L445-L448
https://github.com/kubernetes/kubernetes/blob/release-1.22/hack/local-up-cluster.sh#L593-L597
https://github.com/kubernetes/kubernetes/blob/release-1.22/hack/local-up-cluster.sh#L666

Perhaps only using the public key for the apiserver would be a best practice, but upstream doesn't currently follow it, and obviously has code to handle using the private key for both flags.

[brandond]

The text on that page was suggested by a community member at kubernetes/website#3166 (comment) and perhaps doesn't reflect reality, as the kube-apiserver docs for that flag say:

--service-account-key-file strings
File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key is provided.

So it would appear that it is absolutely valid to pass the private key to both.

[mazzy89]

You are right. According the API server doc is valid to pass a private key. Thanks make sense. I should change some modifications to the website because seems a bit off.

[TheOnlyWei]

@brandond Since the way signing keys work is different from default K8s, is there official confirmation that since the service.key file can use multiple private keys that the first one is used to sign future keys, but every key in the file is used to verify existing tokens?

[brandond]

It's not the same as kubeadm, but everything else is the same as upstream etcd and Kubernetes.

[TheOnlyWei]

Do you know if k3s certificate rotate-ca --path=/opt/k3s/server rotation code simply just updates the, for example, the service account signing key files specified by:

--service-account-key-file
--service-account-signing-key-file
--service-account-private-key-file

I am thinking of implementing our own specific way of rotating keys, but manually updating the files specified in the above arguments, to match the way we are also rotating default K8s keys, so that we can use the same code for both K8s and K3s to rotate service account signing keys.

[brandond]

Please check the docs. Rotate-ca just moves new content into place, it does not generate any new key material or certificates. You can take a look at the provided example scripts to see how new keys and such are actually created.

[TheOnlyWei]

I guess my confusion is that the several places document only private keys usage in service.key. But it is not clear whether you may use public keys in that file.

1. The official sample script rotate-default-ca-certs.sh generates and uses only a private key in service.key as an example.
2. The documentation also uses a private key as an example.
3. This comment in the code implies that at least the first key has to be a private key. But potentially the following keys can be public keys?

   ...The first key in the file should be the current key used to sign ServiceAccount tokens...

However, taking a look at the K3s validateServiceKey code, it uses the public keys of either the public or private keys in the service.key file for validation.

So, I wanted to get an answer from K3s owners if K3s officially supports using public key in service.key file. I haven't been able to find any explicit statements on the expected behavior of service.key, and I don't want to assume the behavior based on circumstantial evidence or the code.

Default K8s supports using public keys through configuration of the private key file (--service-account-signing-key-file) separately from the (per good practices) public key file (--service-account-key-file). Can you do that as well in K3s?

If public keys were not supported by K3s in service.key, it would be good to have that documented explicitly somewhere since it's a departure from default K8s. And if public keys are supported in service.key, it can help clarify things by demonstrating it in another example in this document on how a key rotation would work involving both private and public keys.

[brandond]

The first key in service.key is extracted out to service.current.key at startup:

k3s/pkg/daemons/control/deps/deps.go

Lines 667 to 682 in 518276f

	// When rotating the ServiceAccount signing key, it is necessary to keep the old keys in ServiceKey so that
	// old ServiceAccount tokens can be validated during the switchover process. The first key in the file
	// should be the current key used to sign ServiceAccount tokens; others are old keys used for verification
	// only. Create a file containing just the first key in the list, which will be used to configure the
	// signing controller.
	key, err := keyutil.PrivateKeyFromFile(runtime.ServiceKey)
	if err != nil {
	return err
	}
	keyData, err := keyutil.MarshalPrivateKeyToPEM(key)
	if err != nil {
	return err
	}
	return certutil.WriteKey(runtime.ServiceCurrentKey, keyData)

K3s passes the following args to kube-apiserver: --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key

K3s passes the following args to kube-controller-manager: --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key

So, the first key in the file must be a private key, others can be public keys.