Update atmosphere, ocean, and ice post-proceesing jobs to use COMIN/COMOUT #2705

HenryRWinterbottom · 2024-06-21T16:09:46Z

Description

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow atmosphere and ocean and ice post-processing tasks.

JJOB	Script	COMIN/COMOUT
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMIN_ATMOS_MASTER}/${PREFIX}master.grb2${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMIN_ATMOS_MASTER}/${PREFIX}sfluxgrb${fhr3}.grib2`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrbv.0p25.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2.0p25.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.0p25.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.0p25.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2.0p50.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2.0p50.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.0p50.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.0p50.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2.1p0.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2.1p0.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.1p0.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}pgrb2b.1p0.${fhr3}.idx`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}wgne.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_0p50}/${PREFIX}wgne.${fhr3}`
`JGLOBAL_ATMOS_PRODUCTS`	`exglobal_atmos_products.sh`	`${COMOUT_ATMOS_GRIB_1p0}/${PREFIX}wgne.${fhr3}`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMIN_ICE_HISTORY }}/{{ RUN }}.ice.t{{ HOUR }}z.6hr_avg.f{{ FORECAST_HOUR }}.nc`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_HISTORY }}/{{ RUN }}.ice.t{{ HOUR }}z.6hr_avg.f{{ FORECAST_HOUR }}.nc`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_NETCDF }}/{{ RUN }}.ice.t{{ HOUR }}z.native.f{{ FORECAST_HOUR }}.nc`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_GRIB }}/{{ grid }}/{{ RUN }}.ice.t{{ HOUR }}z.{{ grid }}.f{{ FORECAST_HOUR }}.grib2`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_GRIB }}/{{ grid }}/{{ RUN }}.ice.t{{ HOUR }}z.{{ grid }}.f{{ FORECAST_HOUR }}.grib2.idx`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_OCEAN_HISTORY }}/{{ RUN }}.ocean.t{{ HOUR }}z.6hr_avg.f{{ FORECAST_HOUR }}.nc`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_NETCDF }}/{{ RUN }}.ocean.t{{ HOUR }}z.native.f{{ FORECAST_HOUR }}.nc`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_ICE_GRIB }}/{{ grid }}/{{ RUN }}.ocean.t{{ HOUR }}z.{{ grid }}.f{{ FORECAST_HOUR }}.grib2`
`JGLOBAL_OCEANICE_PRODUCTS`	`exglobal_oceanice_products.py`	`{{ COMOUT_OCEAN_GRIB }}/{{ grid }}/{{ RUN }}.ocean.t{{ HOUR }}z.{{ grid }}.f{{ FORECAST_HOUR }}.grib2.idx`

Refs #2451

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

C48_S2SW experiment on RDHPCS Hera

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202103231200             gfsstage_ic                    62186119           SUCCEEDED                   0         1          25.0
202103231200             gfswaveinit                    62186121           SUCCEEDED                   0         1          34.0
202103231200                 gfsfcst                    62186676           SUCCEEDED                   0         1        2256.0
202103231200    gfsatmos_prod_f000-f006                    62188265           SUCCEEDED                   0         1         453.0
202103231200    gfsatmos_prod_f009-f015                    62188404           SUCCEEDED                   0         1         456.0
202103231200    gfsatmos_prod_f018-f024                    62188403           SUCCEEDED                   0         1         455.0
202103231200    gfsatmos_prod_f027-f033                    62188495           SUCCEEDED                   0         1         409.0
202103231200    gfsatmos_prod_f036-f042                    62188496           SUCCEEDED                   0         1         411.0
202103231200    gfsatmos_prod_f045-f051                    62188851           SUCCEEDED                   0         1         405.0
202103231200    gfsatmos_prod_f054-f060                    62188852           SUCCEEDED                   0         1         403.0
202103231200    gfsatmos_prod_f063-f069                    62189027           SUCCEEDED                   0         1         403.0
202103231200    gfsatmos_prod_f072-f078                    62189028           SUCCEEDED                   0         1         400.0
202103231200    gfsatmos_prod_f081-f087                    62189183           SUCCEEDED                   0         1         410.0
202103231200    gfsatmos_prod_f090-f096                    62189186           SUCCEEDED                   0         1         404.0
202103231200    gfsatmos_prod_f099-f105                    62189321           SUCCEEDED                   0         1         402.0
202103231200    gfsatmos_prod_f108-f114                    62189313           SUCCEEDED                   0         1         407.0
202103231200    gfsatmos_prod_f117-f120                    62189460           SUCCEEDED                   0         1         280.0
202103231200              gfstracker                    62189823           SUCCEEDED                   0         1          70.0
202103231200              gfsgenesis                    62189824           SUCCEEDED                   0         1         533.0
202103231200    gfsocean_prod_f006-f018                    62188415           SUCCEEDED                   0         1         115.0
202103231200    gfsocean_prod_f024-f036                    62188526           SUCCEEDED                   0         1          83.0
202103231200    gfsocean_prod_f042-f054                    62188899           SUCCEEDED                   0         1          80.0
202103231200    gfsocean_prod_f060-f072                    62189046           SUCCEEDED                   0         1          82.0
202103231200    gfsocean_prod_f078-f090                    62189197           SUCCEEDED                   0         1          83.0
202103231200    gfsocean_prod_f096-f108                    62189323           SUCCEEDED                   0         1          84.0
202103231200    gfsocean_prod_f114-f120                    62189485           SUCCEEDED                   0         1          59.0
202103231200    gfsice_prod_f006-f018                    62188416           SUCCEEDED                   0         1         101.0
202103231200    gfsice_prod_f024-f036                    62188527           SUCCEEDED                   0         1          61.0
202103231200    gfsice_prod_f042-f054                    62188900           SUCCEEDED                   0         1          59.0
202103231200    gfsice_prod_f060-f072                    62189047           SUCCEEDED                   0         1          62.0
202103231200    gfsice_prod_f078-f090                    62189198           SUCCEEDED                   0         1          63.0
202103231200    gfsice_prod_f096-f108                    62189324           SUCCEEDED                   0         1          64.0
202103231200    gfsice_prod_f114-f120                    62189486           SUCCEEDED                   0         1          46.0
202103231200          gfswavepostsbs                    62187812           SUCCEEDED                   0         1        2330.0
202103231200          gfswavepostpnt                    62189325           SUCCEEDED                   0         1        7445.0
202103231200                 gfsarch                    62194484           SUCCEEDED                   0         1          46.0
202103231200              gfscleanup

CI/CD will test the additional applications

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

aerorahul · 2024-06-21T16:15:43Z

parm/post/oceanice_products.yaml

@@ -36,18 +36,18 @@ ocean:
  subset: ['SSH', 'SST', 'SSS', 'speed', 'MLD_003', 'latent', 'sensible', 'SW', 'LW', 'LwLatSens', 'Heat_PmE', 'SSU', 'SSV', 'taux', 'tauy', 'temp', 'so', 'uo', 'vo']
  data_in:
    copy:
-      - ["{{ COM_OCEAN_HISTORY }}/{{ RUN }}.ocean.t{{ current_cycle | strftime('%H') }}z.6hr_avg.f{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/ocean.nc"]
+      - ["{{ COMOUT_OCEAN_HISTORY }}/{{ RUN }}.ocean.t{{ current_cycle | strftime('%H') }}z.6hr_avg.f{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/ocean.nc"]


This is in data_in section, so it should be COMIN_OCEAN_HISTORY, no?

Yes, thank you. Testing update and will merge once test passes.

aerorahul · 2024-06-21T17:33:33Z

parm/post/oceanice_products.yaml

@@ -60,16 +60,16 @@ ice:
  subset: ['hi_h', 'hs_h', 'aice_h', 'Tsfc_h', 'uvel_h', 'vvel_h', 'frzmlt_h', 'albsni_h', 'mlt_onset_h', 'frz_onset_h']
  data_in:
    copy:
-      - ["{{ COM_ICE_HISTORY }}/{{ RUN }}.ice.t{{ current_cycle | strftime('%H') }}z.6hr_avg.f{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/ice.nc"]
+      - ["{{ COMOUT_ICE_HISTORY }}/{{ RUN }}.ice.t{{ current_cycle | strftime('%H') }}z.6hr_avg.f{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/ice.nc"]


Same comment here

Thanks. Got it following the previous comment.

aerorahul · 2024-06-21T17:34:55Z

@HenryWinterbottom-NOAA There is not a single COMIN in the table? Where is the data for generating the products coming from?

HenryRWinterbottom · 2024-06-21T17:36:43Z

That table was generated prior to the original PR. It will be updated following addressing your comments and the tests pass.

WalterKolczynski-NOAA · 2024-06-24T06:29:58Z

That table was generated prior to the original PR. It will be updated following addressing your comments and the tests pass.

Creating table should be the first step in these PRs, before you make any changes. Then you use it to determine what should be IN and what is OUT.

…ryWinterbottom-NOAA/global-workflow into feature/gwdev_issue_2451.003

aerorahul · 2024-06-24T18:21:10Z

@HenryWinterbottom-NOAA
In the description of the PR, How has this been tested, it says, C48_ATM on Hera, and then the rocotostat shows gfsocean, gfsice and gfswave jobs run as part of that experiment. How is that possible? C48_ATM only runs the atmosphere.

HenryRWinterbottom · 2024-06-24T18:23:32Z

@HenryWinterbottom-NOAA
In the description of the PR, How has this been tested, it says, C48_ATM on Hera, and then the rocotostat shows gfsocean, gfsice and gfswave jobs run as part of that experiment. How is that possible? C48_ATM only runs the atmosphere.

Typo.

aerorahul · 2024-07-03T16:53:26Z

OBE. Closing.

HenryRWinterbottom and others added 3 commits June 21, 2024 15:32

Bringing branch UTD with WIP.

7bd4962

Merge branch 'NOAA-EMC:develop' into feature/gwdev_issue_2451.003

bd8839c

Added reviewer comment.

d2841cc

HenryRWinterbottom marked this pull request as ready for review June 21, 2024 16:12

HenryRWinterbottom requested review from WalterKolczynski-NOAA and aerorahul June 21, 2024 16:13

aerorahul reviewed Jun 21, 2024

View reviewed changes

Merge branch 'develop' into feature/gwdev_issue_2451.003

ae74746

HenryRWinterbottom and others added 3 commits June 24, 2024 16:42

Updates to indicate inputs to post-processing jobs.

f086cd7

Merge branch 'feature/gwdev_issue_2451.003' of https://github.com/Hen…

9a60526

…ryWinterbottom-NOAA/global-workflow into feature/gwdev_issue_2451.003

Merge branch 'develop' into feature/gwdev_issue_2451.003

e819512

Merge branch 'develop' into feature/gwdev_issue_2451.003

d3e4f3c

aerorahul closed this Jul 3, 2024

HenryRWinterbottom deleted the feature/gwdev_issue_2451.003 branch July 29, 2024 00:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update atmosphere, ocean, and ice post-proceesing jobs to use COMIN/COMOUT #2705

Update atmosphere, ocean, and ice post-proceesing jobs to use COMIN/COMOUT #2705

HenryRWinterbottom commented Jun 21, 2024 •

edited

Loading

aerorahul Jun 21, 2024

HenryRWinterbottom Jun 21, 2024 •

edited

Loading

aerorahul Jun 21, 2024

HenryRWinterbottom Jun 21, 2024

aerorahul commented Jun 21, 2024

HenryRWinterbottom commented Jun 21, 2024

WalterKolczynski-NOAA commented Jun 24, 2024 •

edited

Loading

aerorahul commented Jun 24, 2024

HenryRWinterbottom commented Jun 24, 2024

aerorahul commented Jul 3, 2024

Update atmosphere, ocean, and ice post-proceesing jobs to use COMIN/COMOUT #2705

Update atmosphere, ocean, and ice post-proceesing jobs to use COMIN/COMOUT #2705

Conversation

HenryRWinterbottom commented Jun 21, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Change characteristics

Checklist

aerorahul Jun 21, 2024

Choose a reason for hiding this comment

HenryRWinterbottom Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

aerorahul Jun 21, 2024

Choose a reason for hiding this comment

HenryRWinterbottom Jun 21, 2024

Choose a reason for hiding this comment

aerorahul commented Jun 21, 2024

HenryRWinterbottom commented Jun 21, 2024

WalterKolczynski-NOAA commented Jun 24, 2024 • edited Loading

aerorahul commented Jun 24, 2024

HenryRWinterbottom commented Jun 24, 2024

aerorahul commented Jul 3, 2024

HenryRWinterbottom commented Jun 21, 2024 •

edited

Loading

HenryRWinterbottom Jun 21, 2024 •

edited

Loading

WalterKolczynski-NOAA commented Jun 24, 2024 •

edited

Loading