The overall goal for this CI is to compare checksums generated by the model using a particular config branch/tag against a 'ground truth' checksum.
The Reproducibility CI is comprised of two main triggers: on Pull Request and Scheduled Checks.
There is also CI for modifications to the configuration of the above Reproducibility CI itself (found in config/ci.json) and a user-run workflow for creation of checksums on a given branch.
All of these make use of reusable workflows and custom actions in the ACCESS-NRI/model-config-tests repository.
This Pipeline compares configurations modified in a PR (the source branch) against a 'ground truth' checksum, usually the target branch. It also verifies that commons mistakes in configurations are not made. This allows developers to know if the changes they are about to commit lead to valid and reproducible results. Either way, if the PR is merged, the new commit is tagged in such a way that we know how reproducible it is against past configurations. See the Config Tags section in the main README.md.
This Pipeline runs certain configurations defined in config/ci.json against their already established checksums, as a sort of self-check that validates that we have reproducible results.
This dispatchable workflow allows developers to generate and commit initial checksums to a given branch, usually used in the case where a release-* config has not yet been created.
This is the config/ci.json configuration file for specifying different test markers, or test versions based on type of the test to run, and the name of the git branch or tag. The different types of test are defined as:
scheduled: Scheduled monthly reproducibility tests. The keys under these tests represent released config tags to run scheduled checks on.reproducibility: Reproducibility tests that are run as part of pull requests. The keys under these tests represent the target branches into which pull requests are being merged.qa- Quick quality assurance tests that are run as part of pull requests. The keys under these tests represent the target branches into which pull requests are being merged.
The configuration properties needed to run the tests are
| Name | Type | Description | Example |
|---|---|---|---|
| markers | string |
Markers used for the pytest checks, in the python format | checksum |
| model-config-tests-version | string |
The version of the model-config-tests | 0.0.1 |
| python-version | string |
The python version used to create test virtual environment on Github hosted tests | 3.11.0 |
| payu-version | string |
The Payu version used to run the model | 1.1.4 |
As most of the tests use the same test and python versions, and similar markers, there are two levels of defaults. There's a default at test type level which is useful for defining test markers - this selects certain pytests to run in model-config-tests. There is an outer global default, which is used if a property is not defined for a given branch/tag, and it is not defined for the test default. The parse-ci-config action applies the fall-back default logic. For more information on using this action see ACCESS-NRI/model-config-tests.
The CI for this file (in config.yml) validates modifications to the ci.json against it's schema, found in ACCESS-NRI/schema. It does not yet verify that modifications make sense.