Small changes to get (aerosol) cycling going at C384 #1763

CoryMartin-NOAA · 2023-07-26T19:36:38Z

Description

Trying both ATM C384/C192 cycling with GSI hybrid-4DEnVar (half res of ops) and ATMA C384 3DVar on Hera, I ran into several problems. This PR solves them (for now!)

Change summary:

The aeroanl layouts are now all 8x8, this is just so that we can use the C96 BUMP files that are 8x8 layout for higher resolutions (this is not expected to be a permanent change, and we will have 'optimized' values later)
analcalc would fail (Closes Task gdasanal failed on C384 gdas cycle run #1738 ), this specifies the amount of memory and allows it to complete
The first half cycle C384 forecast would fail, changing nthreads_fv3 to 2 fixes this
The epos jobs do not need 14 nodes allocated, this changes it to 2 nodes, and it runs at this resolution in only 3 mins on Hera
A bugfix (missing f before a f-string) and changed _res_enkf to _res_anl because CASE_ANL is set in config.aeroanl but CASE_ENKF will not be set by the setup_expt script if you specify nmem to be 0

Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Cycled test on Hera ATM
Cycled test on Hera ATMA

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

RussTreadon-NOAA

Did test the aero changes but look OK to me.

WalterKolczynski-NOAA · 2023-07-26T20:00:30Z

parm/config/gfs/config.resources

@@ -885,7 +886,7 @@ elif [[ ${step} = "epos" ]]; then
    export npe_epos=80
    export nth_epos=4
    if [[ "${machine}" == "HERA" ]]; then
-      export nth_epos=6
+      export nth_epos=1


Should this be changed to 1 for all machines? We can get rid of this special case for Hera and just make it 1 everywhere.

Probably but I did not test and am not sure why this was 4 and 6 in the first place... It seems like it is not using any threads at all.

Looks like Kate lowered it to 4 after the WCOSS2 port (PR #1070). And before that it had been lowered to 6 from 8 pre-GitHub. Who knows why it was so high to begin with (I would assume memory). Given Hera is the slowest, lowest-mem system, I suspect lowering it everywhere would work.

With Russ's reply, I'm more confident we can just use 1.

RussTreadon-NOAA · 2023-07-26T20:14:58Z

I checked enkfgdas job log files on WCOSS2. The operational enkfgdas_post jobs run with 4 threads. getsfcensmeanp.x is not threaded. The smoothing sections of getsigensmeanp_smooth.x are threaded. We used to run this code with smoothing on. We do not exercise the smoothing portion of the code in the current operational enkfgdas_post.

I don't know why we have 6 threads for Hera.

WalterKolczynski-NOAA

Conditionally approved pending CI results. I don't expect any issues with the tests, since these changes mainly impact higher resolutions.

aerorahul

The changes are ok with me. Though, I think for the change in config.ufs for C384 should be tested in a coupled forecast.

aerorahul · 2023-07-26T20:27:43Z

parm/config/gfs/config.ufs

@@ -151,7 +151,7 @@ case "${fv3_res}" in
        export layout_y=8
        export layout_x_gfs=8
        export layout_y_gfs=8
-        export nthreads_fv3=1
+        export nthreads_fv3=2


This resolution should be tested more broadly with a S2SW forecast only application.

CoryMartin-NOAA · 2023-07-26T20:47:30Z

Don't merge this in yet even if everything passes, the enkfgdasesfc job fails running out of time, want to make sure my change I just tried allows it to complete. 6 mins was not enough.

emcbot · 2023-07-26T20:57:14Z