Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Leader Elections for VPA Updater and Recommender #6846

Open
lliu8080 opened this issue May 20, 2024 · 6 comments · May be fixed by #6985
Open

Implement Leader Elections for VPA Updater and Recommender #6846

lliu8080 opened this issue May 20, 2024 · 6 comments · May be fixed by #6985
Assignees
@ialidzhikov
Labels
area/vertical-pod-autoscaler kind/feature Categorizes issue or PR as related to a new feature.

Comments

@lliu8080
Copy link

lliu8080 commented May 20, 2024 •
edited
Loading

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

Vertical Pod Autoscaler (VPA) has 3 parts: Recommender, Updater and Admission Controller. Currently only the Admission Controller can run with multiple instances, but not the Recommender and Updater.

It can be problematic to run Recommender and Updater with a single replica as the replica can go into ImagePullBackOff, crashloopbackoff, OOM killed or other bad states after a new release deployment or node patching. Recommender and Updater should support multiple replicas and HA.

We want to enable VPA for large multi-tenant Kubernetes cluster fleet with thousands of EKS clusters. Multiple replicas will help ensure availability and make our lives easier.

Describe the solution you'd like.:
This feature request is to implement leader elections for VPA recommender and updater so VPA recommender and updater can support multiple replicas.

Describe any alternative solutions you've considered.:

Additional context.:
@lliu8080 lliu8080 added the kind/feature Categorizes issue or PR as related to a new feature. label May 20, 2024
@adrianmoisey
Copy link
Contributor

/area vertical-pod-autoscaler

@nikimanoledaki
Copy link
Contributor

nikimanoledaki commented May 29, 2024 •
edited
Loading

This would be a great feature addition. We also run VPA in large Kubernetes clusters so having a HA VPA would be great.

@nikimanoledaki nikimanoledaki mentioned this issue May 29, 2024
Open
@apy-liu
Copy link

apy-liu commented May 29, 2024 •
edited
Loading

Agree this is important for running VPA at a large scale, would be interested to see this get prioritized.

@rolland-zhang
Copy link

Single replicas of critical components for updater/recommender that contribute to managing cluster state always have a higher chance of failure. This is one of the features for VPA our team would be looking forward to the most

@adrianmoisey
Copy link
Contributor

Another thought here: A users normal expectation is to be able to run multiple replicas of an application (I know this was my assumption when starting with the VPA). For a while we ran too many replicas until we figured out that it was not the correct thing to do.
Adding a leader election can help users run the correct number of "active" VPAs (ie: 1).

client-go seems to support Leader Election: https://github.com/kubernetes/client-go/blob/c415c7650f46e6852e429307b5ab4f9435797b74/examples/leader-election/main.go

@ialidzhikov
Copy link
Contributor

/assign

@ialidzhikov ialidzhikov linked a pull request Jun 28, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ialidzhikov ialidzhikov

Labels
area/vertical-pod-autoscaler kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vpa-{updater,recommender}: Add support for leader election ialidzhikov/autoscaler
7 participants
@adrianmoisey @ialidzhikov @lliu8080 @nikimanoledaki @k8s-ci-robot @apy-liu @rolland-zhang