Adds GDAS GEMPAK jobs to rocoto task mesh #2103

HenryRWinterbottom · 2023-11-29T22:47:32Z

Description

This PR addresses issue #1219. Rocoto task jobs/rocoto/gempak.sh was updated to support JGDAS_ATMOS_GEMPAK. Rocoto task jobs/rocoto/gempakmeta.sh was added to support JGFS_ATMOS_GEMPAK_META and JGDAS_ATMOS_GEMPAK_META_NCDC.

Resolves #1219
References #1222 #2073

NOTE This is a replicated PR open to resolve developer mishap with Github interface.

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

The Rocoto XML configuration has been executed as follows.

user@host$: cd /scratch1/NCEPDEV/da/Henry.Winterbottom/trunk/global-workflow.develop.20231117/workflow
user@host$: ./setup_xml.py /scratch1/NCEPDEV/da/Henry.Winterbottom/work/GLOBAL_WORKFLOW/x002_gfsv17_issue_1219/x002_gfsv17_issue_1219.xml

The resulting Rocoto workflow is as follows.

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202103231200             gfsstage_ic                    51952235              QUEUED                   -         0           0.0
202103231200                 gfsfcst                           -                   -                   -         -             -
202103231200            gfspost_f000                    51952236              QUEUED                   -         0           0.0
202103231200            gfspost_f006                    51952237              QUEUED                   -         0           0.0
202103231200            gfspost_f012                    51952238              QUEUED                   -         0           0.0
202103231200                 gfsvrfy                           -                   -                   -         -             -
202103231200             gfsmetpg2g1                           -                   -                   -         -             -
202103231200             gfsmetpg2o1                           -                   -                   -         -             -
202103231200             gfsmetppcp1                           -                   -                   -         -             -
202103231200               gfsgempak                           -                   -                   -         -             -
202103231200           gfsgempakmeta                           -                   -                   -         -             -
202103231200       gfsgempakmetancdc                           -                   -                   -         -             -
202103231200    gfsgempakncdcupapgif                           -                   -                   -         -             -
202103231200      gfsgempakpgrb2spec                           -                   -                   -         -             -
202103231200                 gfsarch                           -                   -                   -         -             -
202103231200              gfscleanup                           -                   -                   -         -             -

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

aerorahul

Looks ok with the exception that the gempakncdcupagif task is not created in the rocoto mesh. This should error out when running setup_xml.py; so wondering why it doesn't?

workflow/applications/gfs_forecast_only.py

workflow/rocoto/gfs_tasks.py

HenryRWinterbottom · 2023-12-01T16:03:07Z

Looks ok with the exception that the gempakncdcupagif task is not created in the rocoto mesh. This should error out when running setup_xml.py; so wondering why it doesn't?

Should this be addressed in a different issue?

KateFriedman-NOAA

Appear to be missing the npoess_pgrb2_0p5deg job but otherwise the changes to add the other gempak jobs into the rocoto mesh look good. See inline comment with information on that job, which is a task dependency for the pgrb2_spec job.

There will be more work to test and troubleshoot these jobs but that is out-of-scope for this PR.

workflow/rocoto/gfs_tasks.py

KateFriedman-NOAA · 2023-12-01T18:43:12Z

workflow/rocoto/gfs_tasks.py

+
+    def gempakpgrb2spec(self):
+        deps = []
+        dep_dict = {'type': 'metatask', 'name': f'{self.cdump}post'}


This should have a job dependency on the npoess_pgrb2_0p5deg job.

This still needs to be updated. :)

HenryRWinterbottom · 2023-12-05T15:16:48Z

@aerorahul @KateFriedman-NOAA @WalterKolczynski-NOAA

Thank you for taking the time to review this PR. After all of my botches last week, I believe that this PR has updated all of the comments, issues, etc., that are noted above. Please let me know otherwise. If nothing seems to be incorrect, this PR should be read to review (again). Thank you for your patience.

KateFriedman-NOAA

Most prior comments addressed, few more to address. See new comments. Thanks @HenryWinterbottom-NOAA !

KateFriedman-NOAA · 2023-12-05T15:45:52Z

workflow/applications/gfs_cycled.py

        if self.do_npoess:
-            gfs_tasks += ['npoess']
+            gfs_tasks += ['npoess_pgrb2_0p5deg']


The npoess_pgrb2_0p5deg job is part of the gempak family of jobs so you can move this line:

gfs_tasks += ['npoess_pgrb2_0p5deg']

...up under the if self.do_gempak: block above and do away with do_npoess flag wherever it is set/used.

KateFriedman-NOAA · 2023-12-05T15:46:54Z

workflow/rocoto/tasks.py

                   'waveawipsbulls', 'waveawipsgridded', 'wavegempak', 'waveinit',
                   'wavepostbndpnt', 'wavepostbndpntbll', 'wavepostpnt', 'wavepostsbs', 'waveprep',
-                   'npoess']
+                   'npoess_pgrb2_0p5deg']


This is a member of the gempak job family so you can group it with the other gempak jobs.

KateFriedman-NOAA · 2023-12-05T16:09:32Z

workflow/rocoto/gfs_tasks.py

+
+    def gempakpgrb2spec(self):
+        deps = []
+        dep_dict = {'type': 'metatask', 'name': f'{self.cdump}post'}


This still needs to be updated. :)

WalterKolczynski-NOAA · 2023-12-05T16:11:20Z

@HenryWinterbottom-NOAA Kate pointed out an incorrect dependency for one of the jobs that should be corrected, plus there is now a conflict from the changes in your AWIPS PR.

HenryRWinterbottom · 2023-12-05T16:16:58Z

@WalterKolczynski-NOAA I am going to close this PR and open a new that addresses all of the comments. My branch has become broken and convoluted and I can no longer keep up-to-date with the develop branch and maintain all of my additions. I will keep this open such that I have the comments but close once I open the new PR. Sorry for the inconvenience this causes.

HenryRWinterbottom marked this pull request as ready for review November 29, 2023 22:52

HenryRWinterbottom requested review from WalterKolczynski-NOAA, aerorahul and KateFriedman-NOAA November 29, 2023 22:52

HenryRWinterbottom self-assigned this Nov 29, 2023

HenryRWinterbottom added the feature New feature or request label Nov 29, 2023

HenryRWinterbottom mentioned this pull request Nov 30, 2023

add variable tob in diag_table #2105

Merged

7 tasks

aerorahul requested changes Dec 1, 2023

View reviewed changes

workflow/applications/gfs_forecast_only.py Outdated Show resolved Hide resolved

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved

KateFriedman-NOAA reviewed Dec 1, 2023

View reviewed changes

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved

KateFriedman-NOAA reviewed Dec 1, 2023

View reviewed changes

KateFriedman-NOAA requested changes Dec 5, 2023

View reviewed changes

HenryRWinterbottom closed this Dec 5, 2023

HenryRWinterbottom force-pushed the feature/gfsv17_issue_1219 branch from 91b6fff to e62a3a7 Compare December 5, 2023 16:14

HenryRWinterbottom mentioned this pull request Dec 5, 2023

Adds GDAS GEMPAK jobs to rocoto task mesh #2126

Closed

7 tasks

HenryRWinterbottom deleted the feature/gfsv17_issue_1219 branch December 7, 2023 16:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds GDAS GEMPAK jobs to rocoto task mesh #2103

Adds GDAS GEMPAK jobs to rocoto task mesh #2103

HenryRWinterbottom commented Nov 29, 2023

aerorahul left a comment

HenryRWinterbottom commented Dec 1, 2023

KateFriedman-NOAA left a comment

KateFriedman-NOAA Dec 1, 2023

KateFriedman-NOAA Dec 5, 2023

HenryRWinterbottom commented Dec 5, 2023

KateFriedman-NOAA left a comment

KateFriedman-NOAA Dec 5, 2023

KateFriedman-NOAA Dec 5, 2023

KateFriedman-NOAA Dec 5, 2023

WalterKolczynski-NOAA commented Dec 5, 2023

HenryRWinterbottom commented Dec 5, 2023

Adds GDAS GEMPAK jobs to rocoto task mesh #2103

Adds GDAS GEMPAK jobs to rocoto task mesh #2103

Conversation

HenryRWinterbottom commented Nov 29, 2023

Description

Type of change

Change characteristics

Checklist

aerorahul left a comment

Choose a reason for hiding this comment

HenryRWinterbottom commented Dec 1, 2023

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA Dec 1, 2023

Choose a reason for hiding this comment

KateFriedman-NOAA Dec 5, 2023

Choose a reason for hiding this comment

HenryRWinterbottom commented Dec 5, 2023

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA Dec 5, 2023

Choose a reason for hiding this comment

KateFriedman-NOAA Dec 5, 2023

Choose a reason for hiding this comment

KateFriedman-NOAA Dec 5, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Dec 5, 2023

HenryRWinterbottom commented Dec 5, 2023