Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ocean resolution to setup_expt invocation and retire/reduce COMROT/ROTDIR usage #2214

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b78d31c
Update ROTDIR references in documentation
KateFriedman-NOAA Jan 4, 2024
9163c23
Update hosts files to remove COMROOT
KateFriedman-NOAA Jan 4, 2024
db773c9
Replace COMROT with ROTDIR in comment
KateFriedman-NOAA Jan 4, 2024
a7749af
Replace COMROT/comrot with ROTDIR/comroot in setup
KateFriedman-NOAA Jan 4, 2024
48d36aa
Replace rotdir with comroot in README
KateFriedman-NOAA Jan 4, 2024
84d4672
Update ROTDIR usage in configs
KateFriedman-NOAA Jan 4, 2024
2cab192
Replace COMROT with ROTDIR in ci system
KateFriedman-NOAA Jan 5, 2024
2e5fa57
add option to provide an ocean resolution during setup_expt.py
aerorahul Jan 8, 2024
af6d038
Merge branch 'develop' into feature/setup_expt_oceanres
aerorahul Jan 8, 2024
2fc2bb5
Merge branch 'develop' into feature/setup_expt_oceanres
aerorahul Jan 8, 2024
740ef66
fix pynorm error
aerorahul Jan 8, 2024
11c7206
Merge branch 'develop' into feature/setup_expt_oceanres
aerorahul Jan 8, 2024
890efe1
Merge branch 'develop' into feature/setup_expt_oceanres
aerorahul Jan 9, 2024
52876b9
update yamls for ci tests
aerorahul Jan 9, 2024
f8da865
update documentation instances of resdet and resens; add resdetocean
aerorahul Jan 9, 2024
31502f4
update default resdetocean
aerorahul Jan 9, 2024
631eb05
Merge branch 'develop' into feature/setup_expt_oceanres
aerorahul Jan 10, 2024
9dd1193
Merge branch 'NOAA-EMC:develop' into feature/clean_out_rotdir
KateFriedman-NOAA Jan 10, 2024
f6bb1d8
Updated some scripts for COM directories
KateFriedman-NOAA Jan 10, 2024
ab03121
Update jobs/JGLOBAL_ATMOS_POST_MANAGER
KateFriedman-NOAA Jan 10, 2024
15845c3
Update docs/doxygen/mainpage.h
aerorahul Jan 10, 2024
5b67aed
Change rotdir/ROTDIR to comroot/COMROOT in CI tests
KateFriedman-NOAA Jan 11, 2024
236f1a5
Change ROTDIR to COMROOT in CI check script
KateFriedman-NOAA Jan 11, 2024
731076e
Merge PR #2204 branch into PR #2214 branch
KateFriedman-NOAA Jan 11, 2024
19131b6
Update ci/cases/pr/C96_atm3DVar.yaml
aerorahul Jan 11, 2024
3c2c0eb
Merge remote-tracking branch 'upstream/feature/setup_expt_oceanres' i…
KateFriedman-NOAA Jan 11, 2024
5dc7b1f
Merge branch 'NOAA-EMC:develop' into feature/clean_out_rotdir
KateFriedman-NOAA Jan 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ci/cases/pr/C48_ATM.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdet: 48
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 48
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
idate: 2021032312
edate: 2021032312
Expand Down
5 changes: 3 additions & 2 deletions ci/cases/pr/C48_S2SW.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: S2SW
resdet: 48
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 48
resdetocean: 5.0
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
idate: 2021032312
edate: 2021032312
Expand Down
7 changes: 4 additions & 3 deletions ci/cases/pr/C48_S2SWA_gefs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: S2SWA
resdet: 48
resens: 48
resdetatmos: 48
resdetocean: 5.0
resensatmos: 48
nens: 2
gfs_cyc: 1
start: cold
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
idate: 2021032312
edate: 2021032312
Expand Down
7 changes: 4 additions & 3 deletions ci/cases/pr/C96C48_hybatmDA.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdet: 96
resens: 48
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 96
resdetocean: 5.0
resensatmos: 48
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C96C48
idate: 2021122018
Expand Down
6 changes: 3 additions & 3 deletions ci/cases/pr/C96_atm3DVar.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdet: 96
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 96
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
icsdir: ${ICSDIR_ROOT}/C96C48
icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C96C48
idate: 2021122018
edate: 2021122106
nens: 0
Expand Down
7 changes: 4 additions & 3 deletions ci/cases/weekly/C384C192_hybatmda.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdet: 384
resens: 192
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 384
resdetocean: 0.25
resensatmos: 192
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C384C192
idate: 2023040118
Expand Down
5 changes: 3 additions & 2 deletions ci/cases/weekly/C384_S2SWA.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: S2SWA
resdet: 384
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 384
resdetocean: 0.5
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
idate: 2016070100
edate: 2016070100
Expand Down
7 changes: 4 additions & 3 deletions ci/cases/weekly/C384_atm3DVar.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdet: 384
resens: 192
comrot: {{ 'RUNTESTS' | getenv }}/COMROT
resdetatmos: 384
resdetocean: 0.25
resensatmos: 192
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C384C192
idate: 2023040118
Expand Down
2 changes: 1 addition & 1 deletion ci/platforms/gefs_ci_defaults.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
defaults:
!INC {{ HOMEgfs }}/parm/config/gefs/yaml/defaults.yaml
base:
ACCOUNT: ${SLURM_ACCOUNT}
ACCOUNT: {{ 'SLURM_ACCOUNT' | getenv }}
2 changes: 1 addition & 1 deletion ci/platforms/gfs_defaults_ci.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
defaults:
!INC {{ HOMEgfs }}/parm/config/gfs/yaml/defaults.yaml
base:
ACCOUNT: ${SLURM_ACCOUNT}
ACCOUNT: {{ 'SLURM_ACCOUNT' | getenv }}
2 changes: 1 addition & 1 deletion ci/scripts/check_ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ for pr in ${pr_list}; do
if [[ "${num_done}" -eq "${num_cycles}" ]]; then
#Remove Experment cases that completed successfully
rm -Rf "${pslot_dir}"
rm -Rf "${pr_dir}/RUNTESTS/COMROT/${pslot}"
rm -Rf "${pr_dir}/RUNTESTS/COMROOT/${pslot}"
rm -f "${output_ci_single}"
# echo "\`\`\`" > "${output_ci_single}"
DATE=$(date +'%D %r')
Expand Down
4 changes: 2 additions & 2 deletions ci/scripts/run-check_ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ pslot=${2:-${pslot:-?}} # Name of the experiment being tested by this scr

# TEST_DIR contains 2 directories;
# 1. HOMEgfs: clone of the global-workflow
# 2. RUNTESTS: A directory containing EXPDIR and COMROT for experiments
# 2. RUNTESTS: A directory containing EXPDIR and COMROOT for experiments
# # e.g. $> tree ./TEST_DIR
# ./TEST_DIR
# ├── HOMEgfs
# └── RUNTESTS
# ├── COMROT
# ├── COMROOT
# │   └── ${pslot}
# └── EXPDIR
# └── ${pslot}
Expand Down
18 changes: 10 additions & 8 deletions docs/doxygen/mainpage.h
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
With the FV3GFS system, a Rocoto driven workflow is being used. This workflow builds on the experiences and work of Kate.Howard and Terry.McGuinness for the GSM, but strips down a lot of complications arising from the use of a cumbersome <b>\c para_config</b>. The <b>\c para_config</b> no longer exists in the workflow and instead a number of <b> \c config </b> files are added, one each for an individual task e.g. <b>\p config.anal</b> contains the analysis specific information. A base config called as <b>\c config.base</b> contains
information related to the machine, super-structure, etc. The idea behind splitting the <b>\p para_config</b> into individual, smaller and managable configs is to provide a means to run any chosen task without the overhead of the full cycling framework. All the configs are located under <b>\c fv3gfs/config</b>

Additionally, the structure of the <b>\c COMROT</b> directory is now modified to look like operations. This enables the use of the workflow much closer to the operational environment, with the exception of the workflow manager.
Additionally, the structure of the <b>\c ROTDIR</b> directory is now modified to look like operations. This enables the use of the workflow much closer to the operational environment, with the exception of the workflow manager.

This is a very much a work in progress and any issues should be reported back and are greatly appreciated.

Expand All @@ -23,12 +23,12 @@ To setup an experiment, a python script <b>\c setup_expt.py</b> (located in <b>\
$> setup_expt.py -h
usage: setup_expt.py [-h] --pslot PSLOT
[--configdir CONFIGDIR] [--idate IDATE] [--icsdir ICSDIR]
[--resdet RESDET] [--resens RESENS] [--comrot COMROT]
[--resdetatmos RESDET] [--resensatmos RESENS] [--comroot COMROOT]
[--expdir EXPDIR] [--nens NENS] [--cdump CDUMP]

Setup files and directories to start a GFS parallel. Create EXPDIR, copy
config files Create COMROT experiment directory structure, link initial
condition files from $ICSDIR to $COMROT
config files Create ROTDIR experiment directory structure, link initial
condition files from $ICSDIR to $ROTDIR

optional arguments:
-h, --help show this help message and exit
Expand All @@ -40,11 +40,13 @@ To setup an experiment, a python script <b>\c setup_expt.py</b> (located in <b>\
(default: 2016100100)
--icsdir full path to initial condition directory
(default: /scratch4/NCEPDEV/da/noscrub/Rahul.Mahajan/ICS)
--resdet resolution of the deterministic model forecast
--resdetatmos atmosphere resolution of the deterministic model forecast
(default: 384)
--resens resolution of the ensemble model forecast
--resdetocean ocean resolution of the deterministic model forecast
(default: 0. [determined automatically based on atmosphere resolution])
--resensatmos resolution of the ensemble model forecast
(default: 192)
--comrot full path to COMROT
--comroot full path to COMROOT, where ROTDIR (COMROOT+PSLOT) will be created
(default: None)
--expdir full path to EXPDIR
(default: None)
Expand All @@ -53,7 +55,7 @@ To setup an experiment, a python script <b>\c setup_expt.py</b> (located in <b>\
--cdump CDUMP to start the experiment
(default: gdas)

The above script creates directories <b>\c EXPDIR</b> and <b>\c COMROT</b>. It will make links for initial conditions from a location provided via the <b>\c --icsdir</b> argument for a chosen resolution for the control <b>\c --resdet</b> and the ensemble <b>\c --resens</b>. Experiment name is controlled by the input argument <b>\c --pslot</b>. The script will ask user input in case any of the directories already exist. It will copy experiment configuration files into the <b>\c EXPDIR</b> from <b>\c CONFIGDIR</b>.
The above script creates directories <b>\c EXPDIR</b> and <b>\c ROTDIR</b>. It will make links for initial conditions from a location provided via the <b>\c --icsdir</b> argument for a chosen resolution for the control <b>\c --resdetatmos</b> and the ensemble <b>\c --resensatmos</b>. Experiment name is controlled by the input argument <b>\c --pslot</b>. The script will ask user input in case any of the directories already exist. It will copy experiment configuration files into the <b>\c EXPDIR</b> from <b>\c CONFIGDIR</b>.

Sample initial conditions for a few resolutions are available at:<br>
<b>Hera:</b> TODO: /path/here/for/initial/conditions<br>
Expand Down
4 changes: 2 additions & 2 deletions docs/source/errors_faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ Example::
**Solution:** set TERM to "xterm" (bash: export TERM=xterm ; csh/tcsh: setenv TERM xterm)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Issue: Directory name change for EnKF folder in COMROT
Issue: Directory name change for EnKF folder in ROTDIR
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Issue:** The EnKF COMROT folders were renamed during the GFS v15 development process to remove the period between "enkf" and "gdas": enkf.gdas.$PDY → enkfgdas.$PDY
**Issue:** The EnKF ROTDIR folders were renamed during the GFS v15 development process to remove the period between "enkf" and "gdas": enkf.gdas.$PDY → enkfgdas.$PDY

**Fix:** Older tarballs on HPSS will have the older directory name with the period between 'enkf' and 'gdas'. Make sure to rename folder to 'enkfgdas.$PDY' after obtaining. Only an issue for the initial cycle.

10 changes: 5 additions & 5 deletions docs/source/init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ The chgres jobs will have a dependency on the data-pull jobs and will wait to ru

5. Check output:

In the config you will have defined an output folder called ``$OUTDIR``. The converted output will be found there, including the needed abias and radstat initial condition files (if CDUMP=gdas). The files will be in the needed directory structure for the global-workflow system, therefore a user can move the contents of their ``$OUTDIR`` directly into their ``$ROTDIR/$COMROT``.
In the config you will have defined an output folder called ``$OUTDIR``. The converted output will be found there, including the needed abias and radstat initial condition files (if CDUMP=gdas). The files will be in the needed directory structure for the global-workflow system, therefore a user can move the contents of their ``$OUTDIR`` directly into their ``$ROTDIR``.

Please report bugs to George Gayno ([email protected]) and Kate Friedman ([email protected]).

Expand Down Expand Up @@ -449,7 +449,7 @@ The warm starts and other output from production are at C768 deterministic and C
What files should you pull for starting a new experiment with warm starts from production?
------------------------------------------------------------------------------------------

That depends on what mode you want to run -- forecast-only or cycled. Whichever mode, navigate to the top of your ``COMROT`` and pull the entirety of the tarball(s) listed below for your mode. The files within the tarball are already in the ``$CDUMP.$PDY/$CYC/$ATMOS`` folder format expected by the system.
That depends on what mode you want to run -- forecast-only or cycled. Whichever mode, navigate to the top of your ``ROTDIR`` and pull the entirety of the tarball(s) listed below for your mode. The files within the tarball are already in the ``$CDUMP.$PDY/$CYC/$ATMOS`` folder format expected by the system.

For forecast-only there are two tarballs to pull

Expand Down Expand Up @@ -489,7 +489,7 @@ Tarballs per cycle:
com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp7.tar
com_gfs_vGFSVER_enkfgdas.YYYYMMDD_CC.enkfgdas_restart_grp8.tar

Go to the top of your ``COMROT/ROTDIR`` and pull the contents of all tarballs there. The tarballs already contain the needed directory structure.
Go to the top of your ``ROTDIR`` and pull the contents of all tarballs there. The tarballs already contain the needed directory structure.

.. _warmstarts-preprod-parallels:

Expand All @@ -514,7 +514,7 @@ Recent pre-implementation parallel series was for GFS v16 (implemented March 202
+ ../$GDATE/gdas_restartb.tar
+ ../$GDATE/enkfgdas_restartb_grp##.tar (where ## is 01 through 08) (note, older tarballs may include a period between enkf and gdas: "enkf.gdas")

* **Where do I put the warm-start initial conditions?** Extraction should occur right inside your COMROT. You may need to rename the enkf folder (enkf.gdas.$PDY -> enkfgdas.$PDY).
* **Where do I put the warm-start initial conditions?** Extraction should occur right inside your ROTDIR. You may need to rename the enkf folder (enkf.gdas.$PDY -> enkfgdas.$PDY).

Due to a recent change in the dycore, you may also need an additional offline step to fix the checksum of the NetCDF files for warm start. See the :ref:`Fix netcdf checksum section <gfsv17-checksum>`.

Expand Down Expand Up @@ -602,5 +602,5 @@ And then on all platforms:

::

cd $COMROT
cd $ROTDIR
for f in $(find ./ -name *tile*.nc); do echo $f; ncatted -a checksum,,d,, $f; done
Loading