Combining PRs for streamlining testing and merging

[aerorahul]

DO NOT MERGE

This PR is not running cleanly without issues on Hera.

Description

This PR combines:

• Update global-workflow and subcomponents to Hera/Rocky 8 partition #2421 - PR for Hera+Rocky 8. Addresses reviewer comments in that PR, and updates hashes to authoritative repo develop where appropriate. Includes some updates that were discovered as part of testing this PR.
• Fix *earc jobs where the number of members isn't a multiple of 10 #2424 - Hotfix PR for ensemble archive jobs
• Modify APP based on RUN #2413 - Supporting multiple APP for different RUN configurations
• Add option to use traditional threading in the UFS #2384 - Traditional/ESMF threading support

Note the TODO for installing fit2obs version 1.1.0 on Orion, Hercules and WCOSS2 and updating run.spack.ver etc.

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

Since Jenkins on Hera is not available due to Java issues, a bunch of CI runs are being run on Hera at:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/

Update on tests:

The C48mx500_3DVarAOWCDA test fails in gdasprepoceanobs for a missing file gdas.t18z.sst_avhrr_ma_l3u.2021032418.nc4.
See log: /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032418/gdasprepoceanobs.log
In the same experiment the gdasanal job failed with the error:

82: [082]correlated_obsmod*upd_varch_(): >>> ERROR <<<  improperly set GSI_BundleErrorCov
82: [082]dropdead(): at 20240324:003122.203(z-000000)

See log:
/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032418/gdasanal.log
The analysis jobs for 3DVar and 4DEnVar run to completion.

In the C96_atmaerosnow test job prepsnowobs.sh, pip is not found in load_ufsda_modules.sh

/scratch1/NCEPDEV/global/Rahul.Mahajan/gwWork.rocky8/consol/ush/load_ufsda_modules.sh: line 52: pip: command not found

See log:
/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96_atmaerosnowDA/logs/2021122018/gdasprepsnowobs.log
pip list was taken out from load_ufsda_modules.sh

prepsnowobs.sh has an incorrect path to iodaconv. This is corrected.
pyiodaconv is no longer built w/ python 3.7. It is hard-wired, and now updated to 3.10.

Offline UPP job for atm analysis atmanlupp as well as ecen jobs are seeing GLIBC errors like below:

+ JGLOBAL_ATMOS_UPP[22]: /scratch1/NCEPDEV/global/Rahul.Mahajan/gwWork.rocky8/consol/scripts/exglobal_atmos_upp.py
python3: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/libpython3.11.so.1.0)
python3: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/libpython3.11.so.1.0)

See log:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96_atm3DVar/logs/2021122100/gdasatmanlupp.log

and

+ exgdas_enkf_ecen.sh[186]: srun -l --export=ALL -n 80 --cpus-per-task=2 /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNDIRS/C96C48_hybatmDA/ecen.190800/getsigensmeanp_smooth.x ./ atminc_ensmean atminc 2
 1: /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNDIRS/C96C48_hybatmDA/ecen.190800/getsigensmeanp_smooth.x: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/hdf5-1.14.0-lixiejp/lib/libhdf5.so.310)

The log:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96C48_hybatmDA/logs/2021122100/enkfgdasecen001.log

These could these be related to stack or the machine, but several other python jobs are running fine as well as other executables. A Hera helpdesk ticket has been opened.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary