Test Clean Up

Test Clean Up

Our regression suite take an average of 4.5 hours to run and fail intermittently. This proposal is to work on different fronts to reduce the intermittent nature of the tests and run the tests in under 30 minutes.

Speeding Up

• The plan is to split the test run into 10 chunks. Each chunk will be run separately (in docker containers for developers and on different machines on CI).
• The results will be aggregated and presented to developers. The details of this are still in flux and will change based on what we discover along the way.
• When a new test is added, we'll have to manually add them to a chunk or we'll have to run a script that will run it and put it in a chunk automatically. Obviously the latter is preferable, but we may need to first get things working.
• Why not CI first? We don't want to break regressions for releases without proper testing. It's more useful to bring the speed up to developers and test out problems it may bring.
• We'll be collecting data on per-test run times so we can re-sort chunks on a regular basis.

Planned Timeline

• Mar: Work on getting tests working locally with Docker first. This will test the chunking logic and bring the benefits of faster testing locally.
• Apr: Get test jobs running on CI running in parallel.
• Jun: Switch the CI jobs to entirely running in parallel.

Reducing Intermittent Nature of Tests

• Add a new flag for tests which lets the job continue after known intermittent tests fail. This is a temporary measure until the test is fixed up. If there's a race condition, we should be fixing it rather than using the flag (needs discussion and buy-in about fixing).
• File a bug for new intermittent failures on a fortnightly basis.

Reducing duplicate Tests

• There was a plan to give our tests a long hard look and eliminate duplicate tests. It's highly likely that we test the same thing in different places. This is worth a good look, but whoever is leading this effort, please add notes and timelines here.