Initial testing framework #1218
Conversation
In order to run test scripts outside of a testing framework, the handling of environment setup should not be solely dependent on running within a dedicated test framework. This has the added benefit of compartmentalizing the duties of environment and dependency solving from running the tests. These environment scripts allow for the selection of a particular environment with the default being the fqdn of the current host. From there, arguments are routed using standard POSIX-sh to a respective script. In the case of Derecho (applicable to any system using lmod) all subsequent argument are treated as modules to load into the current session. The hostenv.sh script relies on one "argument" $AS_HOST being passed in via variable setting to facilitate selection. The helpers.sh script provides convenience features for substing checking in sh, delayed environment variable expansion via eval, and quick banner creation. The derecho.sh script is included as the first supported environment.
This script will facilitate the first tests. There are only three requirements of any given test script with the planned testing framework. If a different testing framework is used in the future, these requirements of the test scripts can and should be re-evaluated. The test script should : 1. Take the intended host / configuration environment as the first argument 2. Take the working directory to immediately change to as the second argument 3. Output some key phrase at the end of the test to denote success, anything else (non-zero exit code, no phrase but return zero) is a failure This particular compilation test script satisfies the above while also providing enough flexibility to select core, target, parallel jobs, and other command-line options into the make build. Additionally, for convenience environment variables can be passed in as command-line options to the test script to modularize certain inputs.
General InstructionsFollow-up details relating to setting up self-hosted runners. Repository Security SettingsFirst, minimal security barriers :
Now you are ready to set up a self-hosted runner. You should still use good PR review techniques to check that no malicious code is present in a PR BEFORE kicking off the workflow. If there are changes in .github/workflows/ or wherever you keep your tests, you should pay extra attention to these changes before allowing a workflow to run. Also consider using label triggers as an extra layer of security to not have workflows automatically start running. This goes counter to the automation process, but is more secure and could potentially save on compute resources especially if many pushes happen in a PR. Creating self-hosted runnersCreating a self-hosted runner is now generally straight forward.
Self-hosted runners are removed from github if they are not connected for a period of time! (At the time of writing this 14 days) https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/removing-self-hosted-runners Runners communication with github : Additional notesI use For runner ids I use Labels I add to runners : I place logs in I name the runners I use For quick setup I have helper scripts available upon request, but I encourage first time setup to be done by hand to understand what is happening. A google doc of this guide with pictures can be found here : https://docs.google.com/document/d/1CJq7NA_bh4ogB37t5Q1m9RO_2XVqPjjdmGQIH4xGWBk/edit?usp=sharing |
|
Would it be possible to place the |
|
That'd be doable, though I'd opt for placing it under |
|
@islas I think the |
Following the documentation of the hpc-workflows testing framework and the testing structure found in .ci/, a JSON file for a GNU compilation test was added. This test will compile the atmosphere core using gnu and single precision. If this test is run using the derecho configuration the appropriate modules will attempt to be loaded. For non-derecho environments, per the testing structure under .ci/, if no configuration exists in .ci/hostenv.sh then the current environment will be used verbatim.
…c-workflows This reusable workflow balances quick setup with github actions-specific features. It assumes that the tests can be controlled via a label being set in a PR. To coordinate PR vs primary branch testing, a suffix is generated using either the PR number or the branch name. This suffix is then used to relocate log files to an archival location in an organized fashion. Github artifacts are still used for failed test capture, but logs will also be moved to the archive location for quicker access if one has access to where these tests execute. To allow for parallelized testing available from hpc-workflows, the workflow can make duplicate directories of the repository that can each run their own test instance without clobbering files. Once tests are run, results are gathered, relocated to archival location, reported and printed to the screen, summarized into the actions summary page, and then packaged into an artifact if failure occured. Finally, the test label is removed if the named tests and label match.
This pipeline is triggered if any pushes occur on master or develop OR if a PR is labeled with an appropriate tag as specified by the tests within this workflow. Additionally, a specific label to trigger all tests can be used that will be removed from the PR when all tests finish, regardless of exit status. The pipeline makes extensive use of the reusable test_workflow.yml to instantiate tests on runners. This pipeline currently only includes the definition for one test to be run on a github runner with tags that satisfy "derecho". Likewise, other hard-coded values appearing in here assume a particular runner setup and environment.
islas
force-pushed
the
initial-testing-framework
branch
from
9a2ff49 to
1b31b47
Compare
|
We should be able to set up runners and test this all out inside this PR before this goes in as well |
During the review of this testing infrastructure into WRF, changes were requested as well as some minor improvements were made. These include: * Update submodule for fixes in filename handling, job naming, and err output * Naming CI/CD jobs with trigger event identifier * Parallelize copy of duplicate directories * Add notes about public repo permissions on actions * Reword env scripts
This PR introduces testing capabilities through a series of compartmentalized commits :
The CI/CD capabilities rely on specific github action runner configurations with the assumption of running on Derecho, but is designed with the intention of minimal reliance on machine and CI/CD-specific tooling. Thus, one could port these features to other machines or CI/CD solutions and achieve similar results.