Unable to authenticate the request. Invalid bearer token, service account token has expired.

[TheOnlyWei]

I'm getting the errors below filling up the K3s.service logs (sudo journalctl --unit=k3s.service -n 100 --no-pager), and I'm wondering what that is caused by and how to resolve it. They seem to be inconsequential to user workloads, but these logs are spammed every few seconds and clutter the logs.

Oct 07 21:53:06 test-vm-ledge k3s[578963]: E1007 21:53:06.842148  578963 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"

I have done a series of service account signing key rotations using the k3s certificate rotate-ca on the cluster. I have also configured service-account-max-token-expiration to 1h00m0s for kube-apiserver-arg:

kube-apiserver-arg:
  - service-account-issuer=<custom OIDC endpoint>
  - service-account-extend-token-expiration=false
  - service-account-max-token-expiration=1h00m0s

We only have one pod in the kube-system namespace, but we do have other pods in other name spaces.

PS C:\Users\testadmin\Desktop> kubectl get pod -n kube-system
NAME                       READY   STATUS    RESTARTS      AGE
coredns-6558475d94-h4r2n   1/1     Running   3 (39d ago)   72d

[brandond]

It's most likely a process running pod that for some reason hasn't reloaded its service account credentials. You can try deleting pods until it goes away, or you could enable apiserver audit logging and identify the denied requests from those.

[TheOnlyWei]

It turns out one of the service's we use has a pod that is failing to call the API server, twice every second and was flooding the K3s.service logs. It seems related to the fact we service-account-max-token-expiration=1h00m0s set, and maybe they are not refreshing the token after one hour.

[brandond]

That's a pretty short value, I imagine a lot of things might expect them to be valid for the default duration.