Change model_data to model in COM templates #2686

WalterKolczynski-NOAA · 2024-06-13T15:53:33Z

What new functionality do you need?

By NCO's request, model_data should be just model in COM paths.

What are the requirements for the new functionality?

model_data replaced by model.

Acceptance Criteria

Output that previous landed in model_data now lands in model
Output is otherwise unchanged

Suggest a solution (optional)

No response

The text was updated successfully, but these errors were encountered:

KateFriedman-NOAA · 2024-06-24T14:47:46Z

@RussTreadon-NOAA @CoryMartin-NOAA FYI about this request from NCO to change model_data subfolder name to just model. I see the following instances of model_data within the gsi_enkf.fd and gdas.cd subcomponents that should be updated to be consistent. I am taking care of the update on the workflow side.

sorc/gsi_enkf.fd/regression/global_enkf.sh:     $nln $datens/$memchar/model_data/atmos/history/${prefix_ens}.atmf00${fhr}.nc sfg_${global_adate}_fhr0${fhr}_${memchar}
sorc/gsi_enkf.fd/regression/global_enkf.sh:            $nln $datens/$memchar/model_data/atmos/history/${prefix_ens}sfcf00${fhr}.nc sfgsfc_${global_adate}_fhr0${fhr}_${memchar}
sorc/gsi_enkf.fd/regression/global_enkf.sh:    $nln $datens/ensstat/model_data/atmos/history/${prefix_ens}.atmf00${fhr}.ensmean.nc sfg_${global_adate}_fhr0${fhr}_ensmean
sorc/gsi_enkf.fd/regression/global_4denvar.sh:dathis=$COMROOTgfs/$dumpges.$PDYg/${cycg}/model_data/atmos/history
sorc/gsi_enkf.fd/regression/global_4denvar.sh:$nln $datens/ensstat/model_data/atmos/history/${prefix_ens}.sfcf006.ensmean.nc         ./sfcf06_anlgrid
sorc/gsi_enkf.fd/regression/global_4denvar.sh:  $nln $datens/$member/model_data/atmos/history/$sigens ${ENS_PATH}sigf${fh}_ens_${member}

sorc/gdas.cd/ush/soca/run_jjobs.py:        model_data = os.path.join(self.com_src, f"{self.RUN}.{self.gPDY}", self.gcyc, "model_data")
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ocean_history_src = os.path.join(model_data, 'ocean', 'history')
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ice_history_src = os.path.join(model_data, 'ice', 'history')
sorc/gdas.cd/ush/soca/run_jjobs.py:        com_ice_restart_src = os.path.join(model_data, 'ice', 'restart')
sorc/gdas.cd/ush/soca/marine_recenter.py:                                       'model_data',
sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py:                                  'model_data', domain, 'history')
sorc/gdas.cd/test/atm/global-workflow/jjob_ens_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_ens_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos/history
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos/history
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos
sorc/gdas.cd/test/atm/global-workflow/jjob_var_init.sh:    source=$GDASAPP_TESTDATA/lowres/$dpath/$memchar/model_data/atmos/history
sorc/gdas.cd/test/snow/apply_jedi_incr.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/create_bkg_ens.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/test_imsproc.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/snow/letkfoi_snowda.sh:RSTDIR=$GDASAPP_TESTDATA/lowres/gdas.$GYMD/$GHR/model_data/atmos/restart
sorc/gdas.cd/test/aero/global-workflow/jjob_var_init.sh:dpath=gdas.$gPDY/$gcyc/model_data/atmos
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ice/history
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ice/restart
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/ocean/history
sorc/gdas.cd/test/soca/gw/prep.sh:mkdir -p ${COM}/06/model_data/atmos/analysis
sorc/gdas.cd/test/soca/gw/prep.sh:cp ${ice_rst} ${COM}/06/model_data/ice/restart/20180415.120000.cice_model.res.nc
sorc/gdas.cd/test/soca/gw/prep.sh:  cp $ocnf ${COM}/06/model_data/ocean/history/gdas.ocean.t06z.inst.f00$i.nc
sorc/gdas.cd/test/soca/gw/prep.sh:  cp $hist_icef ${COM}/06/model_data/ice/history/gdas.ice.t06z.inst.f00$i.nc
sorc/gdas.cd/test/soca/gw/prep.sh:    cp ${COM}/06/model_data/ocean/history/gdas.ocean.t06z.inst.f003.nc \
sorc/gdas.cd/test/soca/gw/prep.sh:    list_of_ocn_fcst=$(ls ${COM}/06/model_data/${domain}/history/gdas.${domain}.t06z.inst.f*.nc)
sorc/gdas.cd/test/soca/gw/prep.sh:    oceandir=${COMENS}/06/mem00${mem}/model_data/ocean/history
sorc/gdas.cd/test/soca/gw/prep.sh:    icedir=${COMENS}/06/mem00${mem}/model_data/ice/history

KateFriedman-NOAA · 2024-06-24T14:48:44Z

@GeorgeGayno-NOAA FYI about this request from NCO to change the model_data subfolder name to just model. I see the following instance of model_data within the UFS_UTILS subcomponent. Please update to be consistent, thanks!

sorc/ufs_utils.fd/util/gdas_init/copy_coldstart_files.sh:  SAVEDIR_MODEL_DATA=$SUBDIR/model_data/atmos/input

RussTreadon-NOAA · 2024-06-24T14:56:25Z

@KateFriedman-NOAA , thank you for letting us know about this NCO requirement. As you note, the g-w changes need to be coordinated with GSI and GDASApp changes. Would you please open a GSI issue and a GDASApp issue to document the required change?

KateFriedman-NOAA · 2024-06-24T15:14:55Z

@RussTreadon-NOAA As requested, issues opened: NOAA-EMC/GSI#761 & NOAA-EMC/GDASApp#1198

GeorgeGayno-NOAA · 2024-06-24T17:34:54Z

@GeorgeGayno-NOAA FYI about this request from NCO to change the model_data subfolder name to just model. I see the following instance of model_data within the UFS_UTILS subcomponent. Please update to be consistent, thanks!
sorc/ufs_utils.fd/util/gdas_init/copy_coldstart_files.sh:  SAVEDIR_MODEL_DATA=$SUBDIR/model_data/atmos/input

Will work this under: ufs-community/UFS_UTILS#967

RussTreadon-NOAA · 2024-06-25T12:14:36Z

NOAA-EMC/GSI has been updated to use model instead of model_data. Done at 6a87460. See GSI PR #762 for details.

RussTreadon-NOAA · 2024-06-25T18:52:44Z

NOAA-EMC/GDASApp:feature/model has been updated to use model instead of model_data. Build and run GDASApp ctests on Orion inside working copy of PR #2700 g-w. All 48 tests pass but notice that model_data remains. This makes sense sinc PR #2700 g-w still uses model_data. See GDASApp issue #1198 for additional information.

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

KateFriedman-NOAA · 2024-06-26T16:20:16Z

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

Hmmmmm, I am considering options for this, one being to merge this work with my issue #2475 work (since I need this work for that issue's work and merging the efforts would reduce work for both of them).

Question for @RussTreadon-NOAA and @GeorgeGayno-NOAA ...are the model_data -> model changes going into your respective develop branches ahead of Orion Rocky9 updates?

RussTreadon-NOAA · 2024-06-26T16:30:09Z

@KateFriedman-NOAA ,

GDASApp

Orion Rocky 9 is already in GDASApp develop. Done via PR #1180
model_data -> model change remains feature/model. PR not opened since change needs to be coordinated with updates to GDASApp test case data directories on WCOSS2, Hera, and Orion/Hercules

GSI

Orion Rocky 9 on pause (see GSI PR #764) given the increased gsi.x and enkf.x wall times
model_data -> model change is already in GSI develop. Done via PR #762

KateFriedman-NOAA · 2024-06-26T16:35:31Z

@RussTreadon-NOAA Ok, given the mixture of order between the model_data -> model changes and Orion Rocky9 updates between the subcomponents I will keep this work separate from the issue #2475 work. I will need to create symlinks within the current IC structure to support the model_data -> model changes going in before my issue #2475 work (which also handles that folder name change but which I don't want to mix with the Orion Rocky9 updates). I'll create the symlinks and create a g-w branch with the workflow side changes. Stay tuned. Thanks!

GeorgeGayno-NOAA · 2024-06-26T17:39:06Z

Do we have a g-w branch with all occurrences of model_data replaced with model? This is the branch in which we should test GDASApp branch feature/model.

Hmmmmm, I am considering options for this, one being to merge this work with my issue #2475 work (since I need this work for that issue's work and merging the efforts would reduce work for both of them).

Question for @RussTreadon-NOAA and @GeorgeGayno-NOAA ...are the model_data -> model changes going into your respective develop branches ahead of Orion Rocky9 updates?

The directory changes were merged yesterday: ufs-community/UFS_UTILS#968

The Orion changes were merged this morning: ufs-community/UFS_UTILS#966

KateFriedman-NOAA · 2024-06-26T17:40:40Z

Noted, thanks @GeorgeGayno-NOAA !

Per a request from NCO Refs NOAA-EMC#2686

AndrewEichmann-NOAA · 2024-07-03T21:23:51Z

Is anybody working on making this change in the template definitions?

KateFriedman-NOAA · 2024-07-08T13:57:21Z

Is anybody working on making this change in the template definitions?

If you're referring to the COM templates in config.com, then yes, I have the updates in a fork branch of mine but have been testing them as part of work for issue #2475. May combine those updates.

Refs NOAA-EMC#2686

This PR converts the staging job from shell to python and introduces the use of yaml. Changes in this PR: 1. Rename `scripts/exglobal_stage_ic.sh` to `scripts/exglobal_stage_ic.py`. 2. Update `jobs/JGLOBAL_STAGE_IC` to use `.py` script extension. Move `COM*` variable declarations and member loop down into yaml and python respectively. Move `GDATE/gPDY/gcyc` settings up to JJOB from ex-script and replace with newer cycle variables (as done in forecast job). 3. Create `parm/stage` folder to hold newly created `stage.yaml.j2`, which both mimics forecast-only functionality in existing `scripts/exglobal_stage_ic.sh` and adds functionality for cycled mode. 4. Create `ush/python/pygfs/task/stage.py` to house staging job python functions for call from `scripts/exglobal_stage_ic.py`. 5. Remove `stage_ic` job rocoto dependencies from xml. Do not need and removes area of duplicate maintenance. 6. Add cycled staging jobs for gdas and enkf suites. 7. Rename `model_data` to `model` for issue #2686 There will now be distinct `stage_ic` jobs for each `RUN`: `gdasstage_ic`, `gfsstage_ic`, `enkfgdasstage_ic`, `stage_ic` (for gefs). Related work was done to set up new symlink folder structure under supported platform `ICSDIR` folder for use by updated staging job. Resolves #2475 Resolves #2650 Resolves #2686 --------- Co-authored-by: Rahul Mahajan <[email protected]> Co-authored-by: Walter Kolczynski - NOAA <[email protected]> Co-authored-by: David Huber <[email protected]>

WalterKolczynski-NOAA added the feature New feature or request label Jun 13, 2024

WalterKolczynski-NOAA assigned KateFriedman-NOAA Jun 20, 2024

This was referenced Jun 24, 2024

Change model_data to model NOAA-EMC/GSI#761

Closed

Change model_data to model NOAA-EMC/GDASApp#1198

Closed

RussTreadon-NOAA mentioned this issue Jun 24, 2024

update path to global ctest model data NOAA-EMC/GSI#762

Merged

4 tasks

GeorgeGayno-NOAA mentioned this issue Jun 24, 2024

Update GDAS init utility for new COM directory ufs-community/UFS_UTILS#967

Closed

GeorgeGayno-NOAA mentioned this issue Jun 25, 2024

Update the gdas_init utility for the new NCO-specified 'model' directory ufs-community/UFS_UTILS#968

Merged

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jun 26, 2024

Rename "model_data" subfolder to "model"

0bebdbf

Per a request from NCO Refs NOAA-EMC#2686

RussTreadon-NOAA mentioned this issue Jun 27, 2024

Update ufs model hash to 20240625 #2729

Merged

7 tasks

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 22, 2024

Change model_data to model

6f5c772

Refs NOAA-EMC#2686

KateFriedman-NOAA mentioned this issue Jul 24, 2024

Convert staging job to python and yaml #2651

Merged

WalterKolczynski-NOAA closed this as completed in #2651 Aug 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change model_data to model in COM templates #2686

Change model_data to model in COM templates #2686

WalterKolczynski-NOAA commented Jun 13, 2024

KateFriedman-NOAA commented Jun 24, 2024

KateFriedman-NOAA commented Jun 24, 2024

RussTreadon-NOAA commented Jun 24, 2024

KateFriedman-NOAA commented Jun 24, 2024

GeorgeGayno-NOAA commented Jun 24, 2024

RussTreadon-NOAA commented Jun 25, 2024

RussTreadon-NOAA commented Jun 25, 2024

KateFriedman-NOAA commented Jun 26, 2024

RussTreadon-NOAA commented Jun 26, 2024

KateFriedman-NOAA commented Jun 26, 2024

GeorgeGayno-NOAA commented Jun 26, 2024

KateFriedman-NOAA commented Jun 26, 2024

AndrewEichmann-NOAA commented Jul 3, 2024

KateFriedman-NOAA commented Jul 8, 2024

Change model_data to model in COM templates #2686

Change model_data to model in COM templates #2686

Comments

WalterKolczynski-NOAA commented Jun 13, 2024

What new functionality do you need?

What are the requirements for the new functionality?

Acceptance Criteria

Suggest a solution (optional)

KateFriedman-NOAA commented Jun 24, 2024

KateFriedman-NOAA commented Jun 24, 2024

RussTreadon-NOAA commented Jun 24, 2024

KateFriedman-NOAA commented Jun 24, 2024

GeorgeGayno-NOAA commented Jun 24, 2024

RussTreadon-NOAA commented Jun 25, 2024

RussTreadon-NOAA commented Jun 25, 2024

KateFriedman-NOAA commented Jun 26, 2024

RussTreadon-NOAA commented Jun 26, 2024

KateFriedman-NOAA commented Jun 26, 2024

GeorgeGayno-NOAA commented Jun 26, 2024

KateFriedman-NOAA commented Jun 26, 2024

AndrewEichmann-NOAA commented Jul 3, 2024

KateFriedman-NOAA commented Jul 8, 2024