Split clean-up into separate job #1906
Merged
Commits on Oct 19, 2023
-
Split clean-up into separate job
Moves the clean-up that was previously done in the archive jobs into their own separate job. It is mostly a copy of what existed in the archive scripts, except some adjustments to account for ensemble members in the same job as the deterministic and some others for shellcheck compliance/other standard code improvements. Some settings used by both jobs were elevated to `config.base`. Others only needed for cleanup were moved to the new config for that job. Resolves NOAA-EMC#583
-
-
-
Rename control variable for cleaning up COM
Since clean-up is no longer part of the archive job, the name of the variable that controls whether COM is cleaned up is renamed. Since the clean-up job does very little if this is NO, in the future may want to just not run this job in that situation. Refs NOAA-EMC#583
-
-
-
The old cleanup script was more convoluted than it needed to be. The script has been streamlined quite a bit and now works properly. Instead of going through every COM template, `COM_TOP is used along with find to get everything at once (well, a few passes of find are needed to deal with regular files, symlinks, and then directories). Refs NOAA-EMC#583 Resolves NOAA-EMC#1872
-
Fix error in rerunning a failed ensemble member
When trying to rerun a failed ensemble member, the script would fail due to `set -u` when grep returned non-zero for not being able to find `PASS` in the group status file. This is resolved by turning it off and then back on.
-
The streamlining of the cleanup script resulted in enkf not being cleaned up. Script not runs for `$RUN` and `enkf${RUN}`. Since the existence of the directory is checked before trying to clean up, it doesn't matter if `enkf${RUN}` is not a valid `$RUN` for the experiment. Refs NOAA-EMC#583 -
-
Add wildcards to cleanup excludes
The `-name` option to find works differently than `grep`, so wild- cards need to be added to make sure files to exclude are matched properly.
-
Split cleanup for enkf off into it's own rocoto job to make the script cleaner. This eliminates the need to loop over RUNs and avoids the unnecessary check of non-existent RUNs. Refs NOAA-EMC#583
-
Use different exclude lists for cleanup
Changes the exclude list into a configurable variable as a comma- separated string. This allows for the use of different strings for ensemble and deterministic. Refs NOAA-EMC#583 Refs NOAA-EMC#1872
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.