Pipeline performance

[joaosilva15]

Hey there 👋 currently while using the pipelines we get some issues during pipeline scheduling. We have the affinity assistant enabled and sometimes multiple affinity assistants get scheduled on the same node making the competing pipeline tasks wait for each other before being able to get scheduled. What I would like to discuss would be how to improve this. I have a couple of ideas but would like to hear opinions first. My ideas would be:

• Lower the base requests for each of our tasks. This would make scheduling easier but it might have an impact on the steps' performance
• Remove the affinity assistant. This would make scheduling easier but it might prevent parallel execution if tasks don't get scheduled on the same node
• Add topologySpreadConstraints to the affinity assistant's template. Currently, only tolerations, nodeSelector, and imagePullSecrets are supported values. With topologySpreadConstraints we could enforce the maximum number of pipelines running in a single node.

What are your thoughts and what do you do in your clusters to make this scheduling issue better?