Add an archive task to GEFS system to archive files in HPSS

[AntonMFernando-NOAA]

Description

• This task is an extension of the arch job previously merged that archives files in ROTDIR (Add an archive task to GEFS system to archive files locally #2816 AntonMFernando-NOAA@2816c3b)

• This feature adds an archive task to GEFS system to archive files in HPSSARCH and LOCALARCH.

  Resolves Add archive task to GEFS workflow #2698
  Refs Add an archive task to GEFS system to archive files locally #2816 Feature/get arch adds an empty archive job to GEFS system #2772 Update archive to handle ensembles #832

Type of change

• new feature

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• CI test in HERA and HERCULES

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[EricSinsky-NOAA]

Here are some initial suggestions.

[emcbot]

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 2 in
/work2/noaa/stmp/CI/HERCULES/2895/RUNTESTS/EXPDIR/C96C48_hybatmDA_ff38f832

[emcbot]

Experiment C96_atm3DVar FAILED on Orion in Build# 3 in
/work2/noaa/stmp/CI/ORION/2895/RUNTESTS/EXPDIR/C96_atm3DVar_ff38f832

[emcbot]

Experiment C96C48_hybatmDA FAILED on Orion in Build# 3 in
/work2/noaa/stmp/CI/ORION/2895/RUNTESTS/EXPDIR/C96C48_hybatmDA_ff38f832

[emcbot]

CI Failed on Hercules in Build# 2
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2895

Experiment C96_atm3DVar_ff38f832 Terminated with 0 tasks failed and 0 dead at Tue Sep 24 15:44:02 CDT 2024
Experiment C96_atm3DVar_ff38f832 Terminated: *STALLED*
Experiment C96C48_hybatmDA_ff38f832 Terminated with 0 tasks failed and 0 dead at Tue Sep 24 15:44:07 CDT 2024
Experiment C96C48_hybatmDA_ff38f832 Terminated: *STALLED*
Experiment C48_ATM_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 17:32:47 CDT 2024
Experiment C48_S2SWA_gefs_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 18:34:17 CDT 2024
Experiment C48_S2SW_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 18:40:00 CDT 2024

[emcbot]

CI Failed on Orion in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/ORION/2895

Experiment C96_atm3DVar_ff38f832 Terminated with 0 tasks failed and 0 dead at Tue Sep 24 04:24:21 PM CDT 2024
Experiment C96_atm3DVar_ff38f832 Terminated: *STALLED*
Experiment C96C48_hybatmDA_ff38f832 Terminated with 0 tasks failed and 0 dead at Tue Sep 24 04:24:23 PM CDT 2024
Experiment C96C48_hybatmDA_ff38f832 Terminated: *STALLED*
Experiment C48_ATM_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 05:37:09 PM CDT 2024
Experiment C48_S2SWA_gefs_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 07:22:46 PM CDT 2024
Experiment C48_S2SW_ff38f832 Completed 1 Cycles: *SUCCESS* at Tue Sep 24 07:45:22 PM CDT 2024

[emcbot]

CI Passed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/2895

Experiment C48mx500_3DVarAOWCDA_fcbb6494 Completed 2 Cycles: *SUCCESS* at Wed Sep 25 00:41:40 UTC 2024
Experiment C48_ATM_fcbb6494 Completed 1 Cycles: *SUCCESS* at Wed Sep 25 01:54:05 UTC 2024
Experiment C48_S2SWA_gefs_fcbb6494 Completed 1 Cycles: *SUCCESS* at Wed Sep 25 02:09:14 UTC 2024
Experiment C48_S2SW_fcbb6494 Completed 1 Cycles: *SUCCESS* at Wed Sep 25 02:44:51 UTC 2024
Experiment C96_atm3DVar_fcbb6494 Completed 3 Cycles: *SUCCESS* at Wed Sep 25 03:39:58 UTC 2024
Experiment C96C48_hybatmDA_fcbb6494 Completed 3 Cycles: *SUCCESS* at Wed Sep 25 06:25:42 UTC 2024
Experiment C96C48_ufs_hybatmDA_fcbb6494 Completed 2 Cycles: *SUCCESS* at Wed Sep 25 07:26:39 UTC 2024
Experiment C96C48_hybatmaerosnowDA_fcbb6494 Completed 3 Cycles: *SUCCESS* at Wed Sep 25 07:26:41 UTC 2024

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_ff38f832 *** SUCCESS *** at 09/24/24 09:42:13 PM
Experiment C48_S2SW_ff38f832 *** SUCCESS *** at 09/24/24 10:07:08 PM
Experiment C96C48_hybatmDA_ff38f832 *** SUCCESS *** at 09/24/24 10:42:29 PM
Experiment C96C48_hybatmaerosnowDA_ff38f832 *** SUCCESS *** at 09/25/24 12:07:26 AM
Experiment C96C48_ufs_hybatmDA_ff38f832 *** SUCCESS *** at 09/25/24 12:56:19 AM
Experiment C96_atm3DVar_extended_ff38f832 *** SUCCESS *** at 09/25/24 08:49:39 AM

[DavidHuber-NOAA]

Apparently I put in a bad memory spec for Orion and Hercules. Debugging.

[DavidHuber-NOAA]

Orion and Hercules are going down for maintenance, so I won't be able to test this. The mem_node_max for both systems are 192GB and 512GB, but it seems that Slurm does not allow requesting a full node's worth of memory. This is similar to Jet and WCCOSS2 where some amount of memory appears to be reserved for the OS. This seems to be a relatively new configuration as these were tested as part of #2727.

I turned both of these down by 12GB and will retest when the system comes back up tomorrow. Since this doesn't affect this PR directly, and all tests passed on Hera and WCCOSS2, I believe this PR is ready to be merged.

[EricSinsky-NOAA]

Tested the most recent additions to the GEFS waves products. Checked the archived products on HPSS and all looks good. Even though the archive job will only run when DO_EXTRACTVARS is YES, this should be ok for the reforecast since DO_EXTRACTVARS will be YES for the reforecast (which is currently the primary target for the GEFS archive task).

addressed changes. needs a re-review if necessary