Deploying your own VPA with leader election enabled in GKE conflicts with the GKE system component

[raywainman]

Which component are you using?:

vertical-pod-autoscaler

What version of the component are you using?:

Component version: vertical-pod-autoscaler-1.2.0+

Leader election functionality was added in #6985 and is turned off by default

What k8s version are you using (kubectl version)?:

Any version 1.27+

What environment is this in?:

GKE

What did you expect to happen?:

The self-deployed VPA recommender and the GKE implementation of HPA to continue working.

What happened instead?:

Both the self-deployed VPA recommender and the GKE version use a lease called vpa-recommender in kube-system.

If you deploy your own VPA recommender, it might "steal" the lease and prevent the GKE implementation of HPA.

How to reproduce it (as minimally and precisely as possible):

• Create a cluster
• Deploy your own vpa-recommender (eg. using vpa-up.sh). Make sure leader election is enabled (leader-elect=true).
• Do something to disrupt the control plane (for example upgrade the version).
• See if your self deployed version of the vpa-recommender has grabbed the lease, if so HPA could stop working.

Anything else?

This is due to the unfortunate naming collision between GKE's system controller (also called vpa-recommender and the one provided here)

[raywainman]

cc @adrianmoisey

Some thoughts on mitigating this:

• Document the flags very carefully to encourage folks not to use the default when deploying on GKE.
• Change the default here which could possibly break some users.

[adrianmoisey]

I see two paths forward to fixing this:

1. GKE stops using vpa-recommender as a lease name
2. VPA stops using vpa-recommender as a lease name

I can't speak for what that lease is being used for in GKE, but I can only assume that changing that lease is difficult or impossible in GKE.

Given that the lease(s) in VPA are only used for VPA components, and running multiple recommenders and updaters for a brief period isn't that worst thing in the world, my vote is that we change the default lease name in the VPA.

Any VPA configured with the lease enabled will only be running multiple pods for a short period of time, which should be fine.

It's obviously not an amazing path forward, but may be worth doing.

I'm curios what @voelzmo and @kwiesmueller think, as they may be the ones approving that controversial PR.

[adrianmoisey]

my vote is that we change the default lease name in the VPA.

If we do go this path, i suggest we also make PRs into 3rd party Helm charts to ensure they support the new default name. Some of them hardcode the lease name:

• https://github.com/cowboysysop/charts/blob/6d2e2b7e82daa507541d5e4e192c26d622624860/charts/vertical-pod-autoscaler/templates/recommender/clusterrole.yaml#L141-L142
• https://github.com/stevehipwell/helm-charts/blob/5f2172e21ce7ba637a78c9717112c7ed27e6586f/charts/vertical-pod-autoscaler/templates/rbac.yaml#L448-L449