Testing limits with ci.centos.org

[ehelms]

• Jenkins node owned by ci.centos.org
  * Request bare metal machines from Duffy
  * Limited to 6/8 parallel machines from Duffy
  * Each OS - install,upgrade pairing requests a machine from Duffy to run a Vagrant pipeline on

  • We end up having 2 jobs rejected when a pipeline runs and it fails

• Will need to scale to additional OSes:

  • Ubuntu 20.04
  • Debian 11
  • CentOS 8 Stream (would eventually replace CentOS 8)

• Proposals

  1. Split our release pipelines similar to the nightly split
     • Schedule EL pipeline, if that succeeds schedule the Debian pipeline
     • Ground work done to enable the split, next step is to split the jobs
       2. Reduce combinations that are run, only a single Debian
       3. Run all installs first, if they pass, run all upgrade jobs
  2. Is there other infrastructure we could explore using available to us?
     • Exploring what IBM cloud might could give us
       • Reasonably priced virt instances
       • Able to run vagrant on top of them
       • Performance comparable to ci.centos.org
     • Exploring Conova supplying HP boxes
     • Exploring the idea of splitting across ci.centos.org and IBM cloud: run Foreman on one, Katello on the other
  3. Look into throttling inside Jenkins
     6. https://plugins.jenkins.io/throttle-concurrents/#example-2-throttling-of-parallel-steps

[ehelms]

Infrastructure SIG notes:

• How do we make this actionable? Where should we be aiming?
• Let's aim to future protect ourselves against running out of resources or blocking multiple pipelines by adding an additional place to run pipelines.
• Potentially Azure FOSS program is a candidate for this as well -- Document Azure's FOSS credits program #1671
• Once we have information about Azure, we will aim to pick a compute provider to split/expand our pipeline workloads to.

[evgeni]

To point out what we discussed in todays SIG meeting:

• this issue is not actionable as such
• we do have a working setup, which has limits, but those won't move anywhere
• CentOS CI might getting a new home, and limits will be different there, but the team is aware of our needs and will try to accommodate these
• Once we actually get new compute targets (like Azure) we can investigate running things there, but this will be part of the (Azure) expansion, not this ticket

→ closed.