Flexibility of config.yaml: a success story

[mkavulich]

It occurs to me that we don't have enough documented "success stories" for when new features are added and they prove useful for specific use cases. So consider this the first entry in that series (and also maybe could provide some examples to include in the Users Guide).

The problem

For our RRFS verification project, I needed a quick way to pull verification observations for a bunch of different dates. I knew this could be done with the verification tasks in the SRW workflow, but the default workflow would have been very clunky to implement: I would have to change the date (for both first and last cycle), then change the working directory so that multiple dates could be pulled simultaneously (pulling data from HPSS often takes dozens of minutes), and also deal with a bunch of unnecessary tasks kicking off.

The solution

I started by copying a verification WE2E test that I knew pulled all observations including snowfall (MET_verification_winter_wx). Since I only wanted to run the observation pulling tasks, I changed my rocoto: section to the following:

rocoto:
  tasks:
    taskgroups: '{{ ["parm/wflow/verify_pre.yaml"]|include }}'
    task_run_MET_Pb2nc_obs:
    metatask_PcpCombine_obs:
    metatask_check_post_output_all_mems:
    metatask_PcpCombine_fcst_APCP_all_accums_all_mems:
    metatask_PcpCombine_fcst_ASNOW_all_accums_all_mems:

Starting with referencing the parm/wflow/verify_pre.yaml file, which contained the observation-pulling tasks, I simply added blank entries for all the other tasks contained in that file, which disabled them. Now I didn't have to worry about unnecessary tasks running, wasting core hours and cluttering up the rocoto task status.

I then changed my workflow: section to include the following settings:

workflow:
  EXPT_SUBDIR: 'test_obs_retrieval_{{ workflow.DATE_FIRST_CYCL }}'
  DATE_FIRST_CYCL: '2023121200'
  DATE_LAST_CYCL: '{{ workflow.DATE_FIRST_CYCL }}'

This meant that I only needed to change one line every time I wanted to change the date, and it would automatically set the EXPT_SUBDIR (the experiment runtime subdirectory) to a unique, descriptive location. No risk of accidentally forgetting to change a line and overwriting a previous run, or forgetting to change DATE_LAST_CYCL and accidentally running a years worth of tasks.

This likely would have required an hour or more of work with the workflow prior to introducing the yaml-specified Rocoto XMLs: it would have required commenting out various parts of a Rocoto XML and making sure there were no undefined variables from those comments, dealing with potentially invalid dependencies. So rather than modifying the Rocoto XML, it would have been way easier to just run the workflow as-is many times, but that would have wasted a lot of core hours, and still would have required more manual changes to the config file every time.

Instead, this took me maybe 10 minutes of work, and I was able to move on to other work (such as writing this post) while I waited for these tasks to retrieve data from HPSS in the background. And again, all this required was a few config.yaml settings.