A few simple questions about the config file

[psooner]

Hi. I have several simple (but possibly dumb) questions about the v2.1.0 config file. [Don't be alarmed by the length here. I'm pretty sure that all/most questions and answers are rather trivial.]

1. Regarding variable PREDEF_GRID_NAME: It's under the "workflow" section in config_defaults.yaml, but it's under the "task_run_fcst" section in the online documentation (Ch. 11). Presumably, I should put it under the "workflow" section, as config_defaults.yaml suggests, right?

2. Regarding variable WTIME_RUN_FCST: The online documentation suggests a default value of "04:30:00", but that variable doesn't appear in config_defaults.yaml. [In fact, none of the WTIME_* variables appear in config_defaults.yaml.] Where are these default values set?

3. Regarding variable FIXshp: Is this a required variable or is it something needed only for plotting? If it's required, will clicking on the SHP link at https://www.naturalearthdata.com/downloads/ provide the necessary files? [If not, would you please provide a link that contains the required files?]

4. Regarding variable EXTRN_MDL_FILES_LBCS: I'll be using GFS files (e.g., gfs.t00z.pgrb2.0p25.f000), so I should literally set EXTRN_MDL_FILES_LBCS to [ gfs.t{hh}z.pgrb2.0p25.f{fcst_hr:03d} ] ? The code will properly handle the {hh} and {fcst_hr:03d} to locate the files that I have pre-staged?

5. Also regarding EXTRN_MDL_FILES_ICS and EXTRN_MDL_FILES_LBCS: In the documentation of EXTRN_MDL_FILES_ICS within config_defaults.yaml, it shows the template enclosed in parentheses () and the use of an equals sign, whereas in the online documentation, it shows the template enclosed in brackets [] (and also using an equals sign). The uncommented-out part of config_defaults.yaml file suggest that a colon should be used to separate the variable name from its value (like all others). I think that I should use a colon to separate the variable name from its value and that I should use brackets [] to enclose the template(s). Is that correct?

6. Regarding variable EXTRN_MDL_DATA_STORES: My external model data will be staged by me. Should I set this to ["disk"], "disk", or something else?

7. Also regarding EXTRN_MDL_DATA_STORES: In the online documentation, this variable appears under two sections: "task_get_extrn_ics" and "task_get_extrn_lbcs". However, in config_defaults.yaml, it occurs only once, under section "platform". I should conform to config_defaults.yaml and put it under the "platform" section, correct?

8. Regarding variables EXTRN_MDL_SYSBASEDIR_ICS/LBCS: Can I ignore these two variables if I am staging my own external model files? [I've set USE_USER_STAGED_EXTRN_FILES: True and I've set appropriate values for EXTRN_MDL_SOURCE_BASEDIR_ICS/LBCS.]

9. The online documentation (section 11.13.7) contains variables TOPO_DIR and SFC_CLIMO_INPUT_DIR, but these are nowhere to be found in config_defaults.yaml. I'm thinking that TOPO_DIR is now (v2.1.0) called FIXorg and SFC_CLIMO_INPUT_DIR is now called FIXsfc. Is this correct?

Thank you for your help.

Phillip S.

[gspetro-NOAA]

Hi Philip,

Happily, most of these questions you've asked should be fairly simple to answer (not dumb though--I had similar types of questions when I first started--there's a bit of a learning curve!)! First though, I want to clarify which documentation you are looking at and which code you are working with. There has been a refactoring of the develop branch code since the v2.1.0 release, and it seems like the code you are looking at may not be matched to the documentation you are working with. What you've written, particularly Questions 1 & 9, suggest that you cloned the develop branch code but are using the v2.1.0 documentation.

Could you tell us what git clone command you used and share the link to the documentation you're working with? You can also try running git branch from your SRW repository and let us know which branch is starred. Then hopefully we can help get everything matched up and answer your questions.

Thanks,
Gillian Petro | Advanced User Support Team

[psooner]

Gillian,

The documentation that I'm using is: https://ufs-srweather-app.readthedocs.io/en/develop/

Section 4.2 of that document says to download the code as such: git clone -b develop https://github.com/ufs-community/ufs-srweather-app.git

git branch shows one entry: * develop

Please let me know what I need to do to get everything matched up (as you said).

Thank you.

Phillip S.

[gspetro-NOAA]

Hi Phillip,

Ok, that explains some of the issue! You're working with develop branch code rather than v2.1.0 code. If you want to work with the v2.1.0 code, the documentation for that is here: https://ufs-srweather-app.readthedocs.io/en/release-public-v2.1.0/ . It is a more stable version of the code, so it will likely be easier to work with, and the documentation for it is fully updated. The develop branch code is always changing, and sometimes the documentation is behind the HEAD of the develop branch. For example, some of the develop branch documentation updates that would help you are waiting for PR #738 to be approved and merged. We can work with you either way. For the v2.1.0 code, I believe the information in questions 1, 2, 3, 7, 9 (and possibly others) would be answered by the v2.1.0 documentation, although I'm happy to provide confirmation if you go that route (as well as providing info on the remaining questions).

If you stick with the develop branch code, then I recommend checking out the info in PR #738 .
1 & 7. All variables in your config.yaml file should appear in the same section that they do in config_defaults.yaml. For example, in Q1, PREDEF_GRID_NAME should go in the workflow: section of your config.yaml file because that's where it currently appears in config_defaults.yaml.
2. Variables that do not appear in config_defaults.yaml can likely be adjusted in one of the ufs-srweather-app/parm/wflow yamls. To adjust the forecast walltime, you'd go to the coldstart.yaml task_run_fcst_mem#mem#: task's walltime: attribute.
3. Plotting information for the develop branch is available here: https://ufs-srweather-app.readthedocs.io/en/develop/RunSRW.html#plotting-configuration-optional The easiest option is to download files from the SRW Data Bucket: https://noaa-ufs-srw-pds.s3.amazonaws.com/NaturalEarth/NaturalEarth.tgz . There is information in PR #738 about additional steps you would have to take with the plotting due to the workflow updates.
9. I believe that is correct! See Issue #231 .

I will be back in a bit with information on the rest once I have a chance to investigate and test it out!

Best,
Gillian

[psooner]

Thank you, Gillian.

I would like to run the stable version of v2.1.0 and have matching documentation.

To do this, would I simply do the git clone command that I did earlier WITHOUT the "-b develop"?
And if I did that, would that code match the documentation link that you provided me?

Thank you.

Phillip S.

[gspetro-NOAA]

Hi Phillip,

There are a few options! In my mind, the easiest is to remove the current clone (run rm -rf ufs-srweather-app from the directory above your ufs-srweather-app directory). Then you can run git clone -b release/public-v2.1.0 https://github.com/ufs-community/ufs-srweather-app.git. If you think you will use the develop branch in the future, you could instead run:

git remote update
git checkout origin/release/public-v2.1.0
git checkout -b release/public-v2.1.0

This pulls in the v2.1.0 code. You will be on the release/public-v2.1.0 branch using v2.1.0 code but have the option to switch to develop at a later time if you want.
Regardless, the v2.1.0 documentation is indeed that link I sent above: https://ufs-srweather-app.readthedocs.io/en/release-public-v2.1.0/

Best,
Gillian

[psooner]

Gillian,

I have downloaded the release/public-v2.1.0 branch and have bookmarked your updated documentation website. Some of my original questions have now been answered. There are a few more that still need answered, please (the ones with "NEED ANSWER").

1. PREDEF_GRID_NAME appears under task_run_fcst according to config_defaults.yaml.

2. The WTIME* variables appear in the updated config_defaults.yaml.

3. FIXshp doesn't appear in the release/public-v2.1.0 branch, so I'm ignoring it.

4. NEED ANSWER

5. NEED ANSWER

6. NEED ANSWER

7. EXTRN_MDL_DATA_STORES occurs under two sections in config_defaults.yaml.

8. NEED ANSWER

9. TOPO_DIR and SFC_CLIMO_INPUT_DIR appear in branch release/public-v2.1.0.

Thank you for your help.

Phillip S.

[natalie-perlin]

Phillip, please see below some responses applicable for the SRW v2.1.0 version, in addition to Gillian's comments about the develop branch

1. Yes, PREDEF_GRID_NAME could to be under "workflow" in your config.yaml files, as it usually remains the same for the entire workflow. Placing it under "task_run_fcst" works as well.

2. WTIME_RUN_FCST: 04:30:00 appears in line 1395 of the config_defaults.yaml in the release/public-v2.1.0 branch, and the other WTIME_. You could set the WTIME_ in config.yaml for your case, overwriting defaults.

3. Shape files are only needed for plotting. FIXshp is not a part of the SRW_v2.1.0

[gspetro-NOAA]

Hi Phillip,

Here are some more answers!

4 & 5. I have not had to set EXTRN_MDL_FILES_LBCS when running the v2.1.0 code. In my experience, as long as EXTRN_MDL_SOURCE_BASEDIR_ICS and EXTRN_MDL_SOURCE_BASEDIR_LBCS point to the directory where your ICS/LBCS files are stored, the system seems to find the data. In my config.yaml file, I set the task_get_extrn_* tasks like this:

task_get_extrn_ics:
  EXTRN_MDL_NAME_ICS: FV3GFS
  FV3GFS_FILE_FMT_ICS: grib2
  USE_USER_STAGED_EXTRN_FILES: true
  EXTRN_MDL_SOURCE_BASEDIR_ICS: /mnt/lfs4/BMC/wrfruc/UFS_SRW_App/v2p1/input_model_data/FV3GFS/grib2/2019061518
task_get_extrn_lbcs:
  EXTRN_MDL_NAME_LBCS: FV3GFS
  LBC_SPEC_INTVL_HRS: 6
  FV3GFS_FILE_FMT_LBCS: grib2
  USE_USER_STAGED_EXTRN_FILES: true
  EXTRN_MDL_SOURCE_BASEDIR_LBCS: /mnt/lfs4/BMC/wrfruc/UFS_SRW_App/v2p1/input_model_data/FV3GFS/grib2/2019061518

For you, the path will correspond to the directory where your files (e.g., gfs.t00z.pgrb2.0p25.f000 and similar) are saved. It should not matter whether file paths are enclosed in double quotes (" ") or not. I've been successful both ways.

However, if you want to set EXTRN_MDL_FILES_LBCS, it seems like you'd set it like this:

task_get_extrn_lbcs:
  EXTRN_MDL_FILES_LBCS:
    - 'gfs.t{hh}z.pgrb2.0p25.f{fcst_hr:03d}'

The code should properly handle the {hh} and {fcst_hr:03d} variables to locate the files that you have pre-staged. You can see an example of how this variable is set in some of the WE2E tests such as this one. EXTRN_MDL_FILES_LBCS seems to be used when getting data from HPSS or if you have data saved in different formats and want the experiment to recognize all of them (then you'd add additional lines starting with -).

6 - You should not need to set EXTRN_MDL_DATA_STORES because it checks disk first, and that is where you are getting your data from. If you want to add it to the config file to be safe, you can add it as "disk" or disk. I've found that using or omitting quotation marks does not seem to change anything for most variables (an exception being single quotes used in template variables).

8 - Correct. You do not need to set EXTRN_MDL_SYSBASEDIR_ICS/LBCS.

In general, I recommend using config_utils.py to check for errors in your config.yaml file after loading the workflow environment (it won't work if the environment hasn't been loaded):
./config_utils.py -c $PWD/config.yaml -v $PWD/config_defaults.yaml

I'm hoping this resolves all of your questions, but let me know if you continue to have any problems! Sometimes answering one question leads to more questions. ;)

Best,
Gillian

[natalie-perlin]

5. Please make sure you are looking at the template for the SRW v2.1.0 (not develop branch).
   The SRW v2.1.0 has the following template in config_defaults.yaml:
   EXTRN_MDL_FILES_LBCS: ""

The "" are just placeholder when no values are set. The actual values could be as following, for example:

EXTRN_MDL_FILES_LBCS:
   - gfs.t12z.pgrb2.0p25.f003
   - gfs.t12z.pgrb2.0p25.f006
   - gfs.t12z.pgrb2.0p25.f009
   - gfs.t12z.pgrb2.0p25.f012
   - gfs.t12z.pgrb2.0p25.f015
   - gfs.t12z.pgrb2.0p25.f018
   - gfs.t12z.pgrb2.0p25.f021
   - gfs.t12z.pgrb2.0p25.f024
   - gfs.t12z.pgrb2.0p25.f027
   - gfs.t12z.pgrb2.0p25.f030
   - gfs.t12z.pgrb2.0p25.f033
   - gfs.t12z.pgrb2.0p25.f036

[psooner]

Gillian and Natalie, thanks for your replies.

Gillian, I'm down to one follow-up question:

In your recent response, you had set
EXTRN_MDL_SOURCE_BASEDIR_ICS: /mnt/lfs4/BMC/wrfruc/UFS_SRW_App/v2p1/input_model_data/FV3GFS/grib2/2019061518

The documentation for EXTRN_MDL_SOURCE_BASEDIR_ICS says: "If USE_USER_STAGED_EXTRN_FILES is set to true, the workflow looks within this directory for a subdirectory named “YYYYMMDDHH”, which contains the external model files..."

So, according to the documentation, I don't need to include the YYYYMMDDHH in EXTRN_MDL_SOURCE_BASEDIR_ICS. Correct?

Thanks.

Phillip S.

[gspetro-NOAA]

Hi Phillip,

I have generally needed to include the YYYYMMDDHH in the EXTRN_MDL_SOURCE_BASEDIR_ICS path. I know there was the hope that the workflow could detect portions of the path from other information in the config file, but as I recall, we ultimately did have to specify that information. I think it also works with ${YYYYMMDDHH} in the path instead of a specific value like 2019061518. Looks like the docs/developers were a tad optimistic. ;) (Good catch!) You could try it without the date to experiment and see if it works, but I think including the date in the path will be most successful.

Best,
Gillian

[psooner]

Thank you, Gillian for all your help!

Phillip S.