[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

DusanJovic-NOAA · 2024-08-23T14:08:49Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

This PR requires two libraries (ESMF v8.6.1 and MAPL v 2.64.3) to be installed on all supported machines. So far, I'm testing these updates on Hercules using ue-esmf-8.6.1-mapl-2.46.2 spack-stack.

Commit Message:

* UFSWM - 
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - 
    * ccpp-physics - 
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * stochastic_physics -

Priority:

Normal

Git Tracking

UFSWM:

Closes Should we update FindESMF.cmake and maybe the name of the imported esmf target #2399

Sub component Pull Requests:

AQM:
CDEPS:
CICE:
CMEPS:
CMakeModules:
FV3:
- ccpp-physics:
- atmos_cubed_sphere:
GOCART:
HYCOM:
MOM6:
NOAHMP:
WW3:
stochastic_physics:
None

UFSWM Blocking Dependencies:

Blocked by #
None

Changes

Regression Test Changes (Please commit test_changes.list):

PR Adds New Tests/Baselines.
PR Updates/Changes Baselines.
No Baseline Changes.

Input data Changes:

None.
New input data.
Updated input data.

Library Changes/Upgrades:

Required
- Library names w/versions:
- Git Stack Issue (JCSDA/spack-stack#)
No Updates

Testing Log:

…ation

bbakernoaa · 2024-09-13T18:43:44Z

@DusanJovic-NOAA If you are testing this with the head of the develop branch from GOCART (hash bf5cf04) then you need to update the tests/parm/gocart/SU2G_instance_SU.rc file

Specifically this line

It will need to be changed with

volcano_srcfilen_explosive: /dev/null
volcano_srcfilen_degassing: /PATH_TO_NEW_FILE/so2_volcanic_emissions_CARN_v202401.degassing_only.rc

I have a copy of this new file on hera: /scratch1/RDARCH/rda-arl-gpu/Barry.Baker/emissions/GEFS/nexus/VOLCANIC/so2_volcanic_emissions_CARN_v202401.degassing_only.rc

Once you have this I think your problems with updating gocart will be solved

DusanJovic-NOAA · 2024-09-17T18:49:06Z

I made the suggested change in SU2G_instance_SU.rc and now I get this error:

$ grep 00000 err
pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=01560    MAPL_EsmfRegridder.F90                   <destination masking with this regrid type is unsupported>
pe=00000 FAIL at line=01382    MAPL_EsmfRegridder.F90                   <status=1>
pe=00000 FAIL at line=00977    MAPL_AbstractRegridder.F90               <status=1>
pe=00000 FAIL at line=00097    NewRegridderManager.F90                  <status=1>
pe=00000 FAIL at line=01101    GriddedIO.F90                            <status=1>
pe=00000 FAIL at line=04539    ExtDataGridCompMod.F90                   <status=1>
pe=00000 FAIL at line=01468    ExtDataGridCompMod.F90                   <status=1>
pe=00000 FAIL at line=01838    MAPL_Generic.F90                         <status=1>
pe=00000 FAIL at line=01241    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=01204    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=01164    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00832    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00972    MAPL_CapGridComp.F90                     <status=1>

bbakernoaa · 2024-09-18T14:56:39Z

@DusanJovic-NOAA do you have the new MAPL/ESMF installed on hera?

DusanJovic-NOAA · 2024-09-18T17:04:44Z

@DusanJovic-NOAA do you have the new MAPL/ESMF installed on hera?

I don't. I already asked EPIC twice to install the updated version of MPAS.

DusanJovic-NOAA · 2024-09-24T16:05:10Z

@bbakernoaa I repeated the cpld_control_p8_intel test on Hercules with updated gocart (top of develop 327ff344) and MAPL 2.46.3 and it fails with the same error:

djovic@hercules-login-4: /work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_4012304/cpld_control_p8_intel
$ grep 00000 err
  0: pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
  0: pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
  0: pe=00000 FAIL at line=01560    MAPL_EsmfRegridder.F90                   <destination masking with this regrid type is unsupported>
  0: pe=00000 FAIL at line=01382    MAPL_EsmfRegridder.F90                   <status=1>
  0: pe=00000 FAIL at line=00977    MAPL_AbstractRegridder.F90               <status=1>
  0: pe=00000 FAIL at line=00097    NewRegridderManager.F90                  <status=1>
  0: pe=00000 FAIL at line=01101    GriddedIO.F90                            <status=1>
  0: pe=00000 FAIL at line=04539    ExtDataGridCompMod.F90                   <status=1>
  0: pe=00000 FAIL at line=01468    ExtDataGridCompMod.F90                   <status=1>
  0: pe=00000 FAIL at line=01838    MAPL_Generic.F90                         <status=1>
  0: pe=00000 FAIL at line=01241    MAPL_CapGridComp.F90                     <status=1>
  0: pe=00000 FAIL at line=01204    MAPL_CapGridComp.F90                     <status=1>
  0: pe=00000 FAIL at line=01164    MAPL_CapGridComp.F90                     <status=1>
  0: pe=00000 FAIL at line=00832    MAPL_CapGridComp.F90                     <status=1>
  0: pe=00000 FAIL at line=00972    MAPL_CapGridComp.F90                     <status=1>

I also updated SU2G_instance_SU.rc to include these two lines:

# Volcanic pointwise sources
volcano_srcfilen_explosive: /dev/null
volcano_srcfilen_degassing: ./so2_volcanic_emissions_CARN_v202401.degassing_only.rc

My run directory on Hercules is /work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_4012304/cpld_control_p8_intel

junwang-noaa · 2024-09-24T16:54:33Z

@weiyuan-jiang Can you access Hercules? May I ask if you can take a look as well?

weiyuan-jiang · 2024-09-24T17:13:41Z

@weiyuan-jiang Can you access Hercules? May I ask if you can take a look as well?

Yes, I will take a look

bena-nasa · 2024-09-25T17:45:17Z

I made the suggested change in SU2G_instance_SU.rc and now I get this error:

$ grep 00000 err
pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=01088    MAPL_CapGridComp.F90                     <status=41>
pe=00000 FAIL at line=01560    MAPL_EsmfRegridder.F90                   <destination masking with this regrid type is unsupported>
pe=00000 FAIL at line=01382    MAPL_EsmfRegridder.F90                   <status=1>
pe=00000 FAIL at line=00977    MAPL_AbstractRegridder.F90               <status=1>
pe=00000 FAIL at line=00097    NewRegridderManager.F90                  <status=1>
pe=00000 FAIL at line=01101    GriddedIO.F90                            <status=1>
pe=00000 FAIL at line=04539    ExtDataGridCompMod.F90                   <status=1>
pe=00000 FAIL at line=01468    ExtDataGridCompMod.F90                   <status=1>
pe=00000 FAIL at line=01838    MAPL_Generic.F90                         <status=1>
pe=00000 FAIL at line=01241    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=01204    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=01164    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00832    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00972    MAPL_CapGridComp.F90                     <status=1>

This crash is not related to the volcanic emissions files. So changing the volcanic emissions files cannot cause this. Either something else was changed or changing the volcanic emissions files just meant the model got further and you hit a different problem.
Could be because of the v2.46.3 update? But how you got to that block is mysterious, that is not the default Regridding option in anything so unless someone changed somehow how you got there is beyond me.

That said, the grid that is getting passed down to gocart and the cap to gocart, does that have a mask set on it somewhere else in UFS?

junwang-noaa · 2024-09-30T14:11:47Z

@DusanJovic-NOAA may I ask where your model source code is for the run /work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_4012304/cpld_control_p8_intel?

DusanJovic-NOAA · 2024-09-30T14:51:23Z

@DusanJovic-NOAA may I ask where your model source code is for the run /work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_4012304/cpld_control_p8_intel?

/work/noaa/fv3-cam/djovic/ufs/gocart_mapl/ufs-weather-model

junwang-noaa · 2024-09-30T16:20:19Z

From @bena-nasa: I have made branch:

https://github.com/GEOS-ESM/MAPL/tree/hotfix/bmauer/candidate_v2.46.4

with what should fix the issue you were seeing with MAPLv2.46.3, please try and let me know. If it fixes your issues we can make a v2.46.4 release of mapl and hotfix this onto our develop.

@RatkoVasic-NOAA would you please install this MAPL version on Hercules for us to test if it resolves the model failure? Thanks

DusanJovic-NOAA added 3 commits August 15, 2024 11:09

Update .gitmodules and CMakeModules

bfd347d

Remove cmake/FindESMF.cmake and rename esmf target to ESMF::ESMF

c994cb0

Special case for finding FindESMF.cmake on WCOSS2 in non-standard loc…

8157915

…ation

DusanJovic-NOAA marked this pull request as draft August 23, 2024 14:09

DusanJovic-NOAA added 2 commits September 16, 2024 14:08

Update FV3, CDEPS and CMEPS

ec4ec28

Merge remote-tracking branch 'origin/develop' into remove_findesmf

a0997d1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

DusanJovic-NOAA commented Aug 23, 2024 •

edited

Loading

bbakernoaa commented Sep 13, 2024

DusanJovic-NOAA commented Sep 17, 2024

bbakernoaa commented Sep 18, 2024

DusanJovic-NOAA commented Sep 18, 2024

DusanJovic-NOAA commented Sep 24, 2024

junwang-noaa commented Sep 24, 2024

weiyuan-jiang commented Sep 24, 2024

bena-nasa commented Sep 25, 2024 •

edited

Loading

junwang-noaa commented Sep 30, 2024

DusanJovic-NOAA commented Sep 30, 2024

junwang-noaa commented Sep 30, 2024

[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

Are you sure you want to change the base?

[DRAFT] Remove local copies of FindESMF from all components, update ESMF to 8.6.1 and MAPL to 2.46.3 #2406

Conversation

DusanJovic-NOAA commented Aug 23, 2024 • edited Loading

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

bbakernoaa commented Sep 13, 2024

DusanJovic-NOAA commented Sep 17, 2024

bbakernoaa commented Sep 18, 2024

DusanJovic-NOAA commented Sep 18, 2024

DusanJovic-NOAA commented Sep 24, 2024

junwang-noaa commented Sep 24, 2024

weiyuan-jiang commented Sep 24, 2024

bena-nasa commented Sep 25, 2024 • edited Loading

junwang-noaa commented Sep 30, 2024

DusanJovic-NOAA commented Sep 30, 2024

junwang-noaa commented Sep 30, 2024

DusanJovic-NOAA commented Aug 23, 2024 •

edited

Loading

bena-nasa commented Sep 25, 2024 •

edited

Loading