Add GEFS atm-only fcst job support #1731

WalterKolczynski-NOAA · 2023-07-10T13:55:32Z

Description

Makes the minor changes necessary to run GEFS atm-only fcst job only from pre-staged ICs. Also many unnecessary DA settings are removed from the GEFS version of config.base. A few settings are retained either because they are needed, or are expected to be needed to support running from reanalysis (e.g. IAU_FHROT).

Resolves #826

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

2+ctrl member GEFS forecast on Orion

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

Makes the minor changes necessary to run GEFS fcst job only from pre-staged ICs. Also many unnecessary DA settings are removed from the GEFS version of `config.base`. Resolves NOAA-EMC#826

emcbot · 2023-07-10T15:10:41Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul 10 13:56:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 14:00:51 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 15:10:25 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:31 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:36 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:39 UTC 2023 for experiment C96_atm3DVar

emcbot · 2023-07-10T15:22:25Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Jul 10 08:56:23 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 08:58:19 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 10:22:08 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:13 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:17 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:22 CDT 2023 for experiment C96C48_hybatmDA

aerorahul · 2023-07-10T15:26:18Z

jobs/JGLOBAL_FORECAST

@@ -17,15 +17,23 @@ rCDUMP=${CDUMP}
 [[ ${CDUMP} = "gfs" ]] && export rCDUMP="gdas"

 # Forecast length for GFS forecast
-if [ ${CDUMP} = "gfs" ]; then
+case ${CDUMP} in
+  gfs | gefs)


In the gefs tree, does FHMAX_GFS still make sense?
Should the line 23 be something like:

current_var=FHMAX_${CDUMP^^}; declare -x FHMAX=${current_var}

And so on.
Beyond this point, there shouldn't be any references to _gfs, _gefs etc.

These kind of things are woven throughout the workflow, well beyond the little portion I am working on here.

parm/config/gefs/config.efcs

jobs/JGLOBAL_FORECAST

emcbot · 2023-07-10T15:57:16Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul 10 13:56:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 14:00:51 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 15:10:25 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:31 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:36 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:39 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 15:57:15 UTC 2023
with 18 successfully completed jobs

The `config.efcs` was using the DA diag tables for GEFS when it should just use the normal diag table from `config.fcst`. Also removed more unneeded DA settings from the GEFS side. Refs: NOAA-EMC#826

emcbot · 2023-07-10T16:06:08Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Jul 10 08:56:23 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 08:58:19 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 10:22:08 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:13 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:17 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:22 CDT 2023 for experiment C96C48_hybatmDA
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 11:06:07 CDT 2023
with 18 successfully completed jobs

emcbot · 2023-07-10T16:45:25Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Mon Jul 10 08:56:23 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 08:58:19 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 10:22:08 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:13 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:17 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 10:22:22 CDT 2023 for experiment C96C48_hybatmDA
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 11:06:07 CDT 2023
with 18 successfully completed jobs
Experiment C96_atm3DVar Terminated: *FAILED*
Experiment C96_atm3DVar Terminated with 1 tasks failed at Mon Jul 10 11:45:20 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1731/RUNTESTS/C96_atm3DVar/COMROT/C96_atm3DVar/logs/2021122100/gdasvrfy.log

TerrenceMcGuinness-NOAA · 2023-07-10T16:52:23Z

@aerorahul said he had a fix for the vrfy file move issue?

WalterKolczynski-NOAA · 2023-07-10T16:57:20Z

@aerorahul said he had a fix for the vrfy file move issue?

His solution was to put the commit hash as part of the pslot.

TerrenceMcGuinness-NOAA · 2023-07-10T17:01:46Z

Oh, I see, didn't know those where related, makes total sense now, ok I"ll test that.

WalterKolczynski-NOAA · 2023-07-10T17:04:17Z

Oh, I see, didn't know those where related, ok I"ll test that.

Well, the path has the pslot in it, and if you use the commit hash that will always be unique unless you rerun a PR without any changes.

That said, given gdasvrfy is now dying the first time running and it isn't a clobber thing, I think something else is going on.

emcbot · 2023-07-10T17:42:16Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul 10 13:56:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 14:00:51 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 15:10:25 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:31 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:36 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:39 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 15:57:15 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Mon Jul 10 17:42:15 UTC 2023
with 148 successfully completed jobs

emcbot · 2023-07-10T18:18:13Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul 10 13:56:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 14:00:51 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 15:10:25 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:31 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:36 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:39 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 15:57:15 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Mon Jul 10 17:42:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Mon Jul 10 18:18:12 UTC 2023
with 86 successfully completed jobs

emcbot · 2023-07-10T18:21:12Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Mon Jul 10 13:56:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Mon Jul 10 14:00:51 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Mon Jul 10 15:10:25 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:31 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:36 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Mon Jul 10 15:10:39 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Mon Jul 10 15:57:15 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Mon Jul 10 17:42:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Mon Jul 10 18:18:12 UTC 2023
with 86 successfully completed jobs

Refs: NOAA-EMC#826

emcbot · 2023-07-11T09:14:47Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Jul 11 07:57:04 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Jul 11 08:02:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Jul 11 09:14:33 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:39 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:43 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:46 UTC 2023 for experiment C96_atm3DVar

emcbot · 2023-07-11T09:57:09Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Jul 11 07:57:04 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Jul 11 08:02:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Jul 11 09:14:33 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:39 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:43 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:46 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Tue Jul 11 09:57:08 UTC 2023
with 18 successfully completed jobs

emcbot · 2023-07-11T11:36:16Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Jul 11 07:57:04 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Jul 11 08:02:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Jul 11 09:14:33 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:39 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:43 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:46 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Tue Jul 11 09:57:08 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Tue Jul 11 11:36:15 UTC 2023
with 148 successfully completed jobs

emcbot · 2023-07-11T12:15:16Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Jul 11 07:57:04 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Jul 11 08:02:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Jul 11 09:14:33 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:39 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:43 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:46 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Tue Jul 11 09:57:08 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Tue Jul 11 11:36:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Tue Jul 11 12:15:16 UTC 2023
with 86 successfully completed jobs

emcbot · 2023-07-11T12:18:14Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Jul 11 07:57:04 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Jul 11 08:02:55 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Jul 11 09:14:33 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:39 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:43 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Jul 11 09:14:46 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Tue Jul 11 09:57:08 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Tue Jul 11 11:36:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Tue Jul 11 12:15:16 UTC 2023
with 86 successfully completed jobs

aerorahul

looks good.

Add GEFS fcst job support

7e8e72a

Makes the minor changes necessary to run GEFS fcst job only from pre-staged ICs. Also many unnecessary DA settings are removed from the GEFS version of `config.base`. Resolves NOAA-EMC#826

WalterKolczynski-NOAA requested a review from aerorahul July 10, 2023 13:55

WalterKolczynski-NOAA self-assigned this Jul 10, 2023

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Jul 10, 2023

WalterKolczynski-NOAA changed the title ~~Add GEFS fcst job support~~ Add GEFS atm-only fcst job support Jul 10, 2023

emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jul 10, 2023

emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Jul 10, 2023

aerorahul reviewed Jul 10, 2023

View reviewed changes

parm/config/gefs/config.efcs Outdated Show resolved Hide resolved

aerorahul reviewed Jul 10, 2023

View reviewed changes

jobs/JGLOBAL_FORECAST Outdated Show resolved Hide resolved

Stop using DA diag table from GEFS, remove unneeded settings

91dfc16

The `config.efcs` was using the DA diag tables for GEFS when it should just use the normal diag table from `config.fcst`. Also removed more unneeded DA settings from the GEFS side. Refs: NOAA-EMC#826

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Jul 10, 2023

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jul 10, 2023

Use RUN instead of CDUMP to determine output frequencies in forecast

b38324e

Refs: NOAA-EMC#826

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Jul 11, 2023

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jul 11, 2023

WalterKolczynski-NOAA requested a review from aerorahul July 11, 2023 14:06

aerorahul approved these changes Jul 11, 2023

View reviewed changes

aerorahul merged commit 84842f4 into NOAA-EMC:develop Jul 11, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GEFS atm-only fcst job support #1731

Add GEFS atm-only fcst job support #1731

WalterKolczynski-NOAA commented Jul 10, 2023 •

edited

Loading

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

aerorahul Jul 10, 2023

WalterKolczynski-NOAA Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

TerrenceMcGuinness-NOAA commented Jul 10, 2023

WalterKolczynski-NOAA commented Jul 10, 2023

TerrenceMcGuinness-NOAA commented Jul 10, 2023 •

edited

Loading

WalterKolczynski-NOAA commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

aerorahul left a comment

Add GEFS atm-only fcst job support #1731

Add GEFS atm-only fcst job support #1731

Conversation

WalterKolczynski-NOAA commented Jul 10, 2023 • edited Loading

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

aerorahul Jul 10, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jul 10, 2023

Choose a reason for hiding this comment

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

TerrenceMcGuinness-NOAA commented Jul 10, 2023

WalterKolczynski-NOAA commented Jul 10, 2023

TerrenceMcGuinness-NOAA commented Jul 10, 2023 • edited Loading

WalterKolczynski-NOAA commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 10, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

emcbot commented Jul 11, 2023

aerorahul left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Jul 10, 2023 •

edited

Loading

TerrenceMcGuinness-NOAA commented Jul 10, 2023 •

edited

Loading