Expanding the SRW Testing Framework

[christinaholtNOAA]

Let's start gathering ideas for better ways to test SRW beyond Workflow End-to-end (WE2E) tests. Please feel free to add any scope or size of test here that you think we need. Even if you don't know how to test a given thing, contribute stories about how you've seen things break that were important for your work.

Here's a list of some areas that have come up in conversation:

• Build tests for of each of the different kinds of builds -- no flags, AQM, and RRFS
• End-to-end tests that test each of those different builds.
• Unit tests should likely be converted to pytest.
• Unit tests should be expanded to maximize coverage.
• Integration and functional tests should be included for each of the utilities in the ush directory (for example, create_*.py). These can be written in Pytest or unittest, but are meant to test how multiple functions work together.
• All python code should be fully linted.
• Linted code should aim to remove many of the "pylint: disable" directives (especially undefined-variables)
• A faster way to test the retrieve_data.py script is needed. Also one that doesn't fail if there is missing data. This could include mocking URLs to tests "pulling data" separately from finding known data locations.
• Additional tasks in WE2E tests that can more readily interrogate WE2E tests and check things like number of files output, length of forecasts, expected tasks in workflow, etc.
• Full regression testing for WE2E tests (check whether the answer changed with a given code modification)

[christinaholtNOAA]

One more came up in today's SRW CM Meeting. Automation of creating standalone wrapper scripts alongside appropriate unit tests and perhaps even WE2E tests, although it's not clear how we might best drive those wrapper-based WE2E tests.

[EdwardSnyder-NOAA]

We are currently testing the wrapper scripts through the Jenkins pipeline using the wrapper_srw_ftest.sh script and is ran whenever we add the run_we2e_comprehensive_tests label to a PR.

[EdwardSnyder-NOAA]

Here are a couple of things that come to my mind in regards to SRW testing:

• Improve the error handling for the MET VX WE2E tests. In the past, the workflow will report successful verification tasks, even when there are MET plus errors. This might have been addressed already. I haven't ran into this issue in some time now.
• There is an "error" when running the retrieve_data.py on platforms that use HPSS as the data source (i.e. Jet and Hera). It appears the retrieve_data.py script checks if HPSS is loaded before it checks for data on disk. I thought this script checks for data on disk first before trying other data sources. Obviously, this isn't an issue if the user loads the hpss module before running the script.

[mkavulich]

I just thought of a cheap way to do a basic "smoke test" of the workflow generation on all platforms. In some updates earlier this year I introduced the flag --launch to the WE2E run script. By running ./run_WE2E_tests.py with the -t all --launch none options, you can run ./generate_FV3LAM_wflow.py for every WE2E test file, ensuring that the generation script completes successfully, without actually running any tests. This will be a very fast (<2 minutes in my tests on Derecho) way to exercise a whole bunch of checks in the workflow generation. And this could potentially be combined with future tests that check that files are created and/or staged correctly.