Flexibility of config.yaml: a success story #993
Closed
mkavulich
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
It occurs to me that we don't have enough documented "success stories" for when new features are added and they prove useful for specific use cases. So consider this the first entry in that series (and also maybe could provide some examples to include in the Users Guide).
The problem
For our RRFS verification project, I needed a quick way to pull verification observations for a bunch of different dates. I knew this could be done with the verification tasks in the SRW workflow, but the default workflow would have been very clunky to implement: I would have to change the date (for both first and last cycle), then change the working directory so that multiple dates could be pulled simultaneously (pulling data from HPSS often takes dozens of minutes), and also deal with a bunch of unnecessary tasks kicking off.
The solution
I started by copying a verification WE2E test that I knew pulled all observations including snowfall (
MET_verification_winter_wx
). Since I only wanted to run the observation pulling tasks, I changed myrocoto:
section to the following:Starting with referencing the
parm/wflow/verify_pre.yaml
file, which contained the observation-pulling tasks, I simply added blank entries for all the other tasks contained in that file, which disabled them. Now I didn't have to worry about unnecessary tasks running, wasting core hours and cluttering up the rocoto task status.I then changed my
workflow:
section to include the following settings:This meant that I only needed to change one line every time I wanted to change the date, and it would automatically set the
EXPT_SUBDIR
(the experiment runtime subdirectory) to a unique, descriptive location. No risk of accidentally forgetting to change a line and overwriting a previous run, or forgetting to changeDATE_LAST_CYCL
and accidentally running a years worth of tasks.This likely would have required an hour or more of work with the workflow prior to introducing the yaml-specified Rocoto XMLs: it would have required commenting out various parts of a Rocoto XML and making sure there were no undefined variables from those comments, dealing with potentially invalid dependencies. So rather than modifying the Rocoto XML, it would have been way easier to just run the workflow as-is many times, but that would have wasted a lot of core hours, and still would have required more manual changes to the config file every time.
Instead, this took me maybe 10 minutes of work, and I was able to move on to other work (such as writing this post) while I waited for these tasks to retrieve data from HPSS in the background. And again, all this required was a few config.yaml settings.
Beta Was this translation helpful? Give feedback.
All reactions