Enable UPP 2D decomposition in the SRW app

[laurenfrederick]

Description

The UPP supports 2D decomposition but this feature cannot be utilized without changes to the SRW app.

[MichaelLueken]

Following the UPP code management meeting last week (08/09/2023), Wen provided me details on how to enable 2d decomposition for both offline and inline post:

For your inquiry of enabling UPP 2D decomposition, here are settings for inline post and offline post:
1) offline post
    Add "numx=$numx"  in the namelist $NAMPGB in itag file at runtime, $number can be set as, 2, 3,.... In UPP code, the default setting of numx is 1.
   for example: decomposite x direction as 2 slices
---------------------------------------------------
    &NAMPGB
KPO=47,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,30.,20.,10.,7.,5.,3.,2.,1.,numx=2,
----------------------------------------------------
2) inline post
Set variable "itasks" in model_configure, such as:
------------------
output_history:          .true.
write_dopost:            .true.
...
itasks: 2
-------------------
The default setting of itasks is 1 for inline post.

The changes were made for offline post and using:

grep numx log/run_post*

showed that numx was being set to 2, which is indicative of 2d decomposition being enabled.

However, when adding itasks: 2 to the model_configure file, there were no prints indicating that itasks was being set to 2, rather than the default of 1. Wen looked into the matter and it appears as though numx was removed from the inline post interface at the I/O directory of the UFS FV3. Hopefully this will be able to be added back in so that users can easily see that they, are in fact, running with 2d decomposition set.

[MichaelLueken]

PR #1823 has successfully been merged to the ufs-weather-model repository. This PR applied updates to FV3/io/post_fv3.F90 so that the value of numx for inline post would be printed. This information is required to see if the 2d decomposition is enabled in the run.

However, this update also includes changes which were merged to the UPP repository in PR #750. This update removed the postxconfig-NT-fv3lam.txt file. Dialogue in the PR suggested replacing postxconfig-NT-fv3lam.txt with postxconfig-NT-fv3lam_rrfs.txt. Simply replacing the post config file isn't enough to make either the inline or offline post to work. Reaching out to Wen, it turns out the postxconfig-NT-fv3lam_rrfs.txt file includes simulated radiances. Moving to this new file requires CRTM coefficient files in order to run. The list of simulated radiances are:

1. imgr_g15
2. imgr_g13
3. imgr_g12
4. imgr_g11
5. amsre_aqua
6. tmi_trmm
7. ssmi_f13
8. ssmi_f14
9. ssmi_f15
10. ssmis_f16
11. ssmis_f17
12. ssmis_f18
13. ssmis_f19
14. ssmis_f20
15. seviri_m10
16. imgr_mt2
17. imgr_mt1r
18. imgr_insat3d
19. abi_gr
20. abi_g16

Once these coefficient files have been added to the exregioanl_run_fcst (for inline post) and exregional_run_post (for offline post) scripts, the fundamental tests are once again passing, with the exception of the grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot test. The new post config file no longer uses PRMSL, but MSLET instead. Due to this change, Pressure reduced to MSL is no longer used to compute SLP, now it is MSLP (Eta model reduction). When this was added to the exregional_plot_allvars(_diff).py scripts, the failing test now passes.

[MichaelLueken]

The necessary changes to UPP 2d decomposition have been made:

• For inline post 2d decomposition, the variable itasks was added to parm/model_configure. The variable was also added to ush/config_defaults.yaml and set to the default value of 1. The ush/create_model_configure_file.py script was updated to include itasks as well. Finally, the config.grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2.yaml configuration file was updated to add itasks: 2 to task_run_fcst. The test was run and the run_fcst log file shows numx = 2, showing that 2d decomposition was used.
• For offline post 2d decomposition, the variable numx was added to the &NAMPGB namelist options in the itag file (which is generated by the exregional_run_post.sh script). The variable was also added to ush/config_defaults.yaml and set to the default value of 1. The grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta configuration file was updated to add numx: 2 to task_run_post. The test was run and the run_post log files shows numx = 2, showing that 2d decomposition was used.

The fundamental tests were run and all passed. The run_post log files were checked the non-updated tests showed numx = 1, not using 2d decomposition.

Finally, the AQM configuration files were updated to add USE_CRTM: false under global, since the AQM post configuration file doesn't require CRTM coefficients to run. The AQM WE2E test was run and successfully passed.

[MichaelLueken]

The fundamental WE2E test suite has been successfully run on Hera Intel, Gaea, and Jet. The ush/machine/*.yaml files have been updated on these machines, as well as Cheyenne.

The feature/upp_2d_decomp branch has been cloned on Derecho. The ush/machine/derecho.yaml file has had the CRTM_DIR variable added to it. Unfortunately, the fundamental tests are failing in the run_post step. Looking in the log files, I see the following:

forrtl: severe (122): invalid attempt to assign into a pointer that is not associated
Image              PC                Routine            Line        Source
upp.x              0000000000BEC955  crtm_forward_modu         356  CRTM_Forward_Module.f90
libiomp5.so        000014AB082CE053  __kmp_invoke_micr     Unknown  Unknown
libiomp5.so        000014AB0823CA64  __kmp_fork_call       Unknown  Unknown
libiomp5.so        000014AB081F6223  __kmpc_fork_call      Unknown  Unknown
upp.x              0000000000BEC3F3  crtm_forward_modu         353  CRTM_Forward_Module.f90
upp.x              000000000075763B  calrad_wcloud_           1725  CALRAD_WCLOUD_newcrtm.f
upp.x              0000000000637E9A  process_                  137  PROCESS.f
upp.x              0000000000582DE0  MAIN__                    699  WRFPOST.f
upp.x              000000000041390D  Unknown               Unknown  Unknown
libc-2.31.so       000014AB03D9729D  __libc_start_main     Unknown  Unknown
upp.x              000000000041383A  Unknown               Unknown  Unknown

Unfortunately, it appears as though there is an issue with the CRTM library on Derecho. Only two of the seven tests have completed thus far, but both have failed in the post with this exact same failure (noted above). Will continue to investigate while wrapping up updates and testing on Orion tomorrow.