Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase Utilization of amd64 Drone Runner #11449

Open
dereknola opened this issue Dec 10, 2024 · 2 comments
Open

Increase Utilization of amd64 Drone Runner #11449

dereknola opened this issue Dec 10, 2024 · 2 comments
Assignees

Comments

@dereknola
Copy link
Member

Is your feature request related to a problem? Please describe.
The amd64 Drone runner used for PRs is a AMD EPYC 7402P, 24/48 Core/Thread processor. When we run the amd64 build pipeline, we utilize less than 25% of the CPU and only 10GB of the 64GB of RAM. The Drone pipelines are currently too sequential to full utilize this resource.

Describe the solution you'd like
Increase parallelization of Drone build steps where possible. Consider:

  • Allowing multiple k3s build pipelines at once
  • Run more E2E tests simultaneously
  • Run more Docker test simultaneously

Describe alternatives you've considered
We have moved most of our workloads to GHA. But these machines are many, small 4vCPU/16GB nodes. Drone runners enable large scale testing and performance, and should be utilized as such.

Additional context

@brandond
Copy link
Member

In the past, increasing test concurrency caused problems with disk IO latency - builds running in parallel with datastore ops and image pulls on the same underlying disk caused problems and we saw a lot of tests running long and flaking. I don't think CPU or memory were the bottleneck.

@dereknola
Copy link
Member Author

Okay, I will do some tests and watch the IO. I was able to successfully run two E2E tests at once. Three at once seems to push the runner too far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: New
Development

No branches or pull requests

2 participants