Salt Toaster's Future

Background

Salt’s in-tree testsuite can be executed on different operating systems for multiple Salt versions
Known test failures can be ignored
Tests that don’t run well can be skipped
suse.test enable cross-distro tests
suse.test enable testing different Salt versions on different Minions connected to a Master with potentially another Salt version
Install RPMs for production-like test environments (dependencies + patched sources)

conftest_source is a mess
KNOWN_ISSUES are not maintained
Image generation is complicated, too many Jinja layers
make as the container entry point is unexpected and not very nice to work with
make is not a good format for people that don’t really know it (same problem we have in Uyuni)
Dependency on saltrepoinspect

Run Salt's in-tree testsuite
- on Salt 3000, 3004, 3004-bundled, and the next Salt version for evaluation (moving target)
- on Pull Requests through GitHub Actions
- on a current version of these Linux distributions
  - Leap
  - CentOS/AlmaLinux
  - Debian
  - Ubuntu
- on older operating system we still support
  - SLE 12
  - CentOS 7
  - Ubuntu 20.04
- locally on a developer’s computer
- on a Jenkins runner
Known failures can be ignored
Broken tests (don’t run for whatever reason) can be skipped
Cross-distro tests (“suse.test”) that run a master, some minions with different Salt versions and operating systems
Test environment can be adjusted/extended easily, also by newcomers
Salt and dependencies are installed from repositories
Modular architecture

Unified CLI for updating the test environment, executing the tests
Configuration file that defines which tests to run on what operating systems etc.
Run tests on changed code only