Add GEFS regression test suite from EP5r2 configuration/case #2442
Conversation
…esn't run with namelist error
| SU_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 SO2 ExtData/QFED_Blended_20210325_50days.nc | ||
| OC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 OC ExtData/QFED_Blended_20210325_50days.nc | ||
| BC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 BC ExtData/QFED_Blended_20210325_50days.nc |
There was a problem hiding this comment.
@aerorahul I believe these are the only custom files for the regression test, so as not to carry so much data on all platforms. ExtData is not staged yet...may be nice to be consistent between RT and workflow
|
@lipan-NOAA @junwang-noaa Intel debug reliably fails in GOCART with traceback to Floating point exception: floating-point divide by zero at GOCART/Process_Library/GOCART2G_Process.F90:1575 on n_atmsteps = 16: I would imagine FV3 would complain first about any vanishing layer thickness (?), from say stochastic physics Interestingly, a 3 hour forecast completes if I change this line to: Maybe there is some haloes or padding to arrays leading to dz=0? I'm not sure where to go from here |
@NickSzapiro-NOAA So you are running the test in debug mode? It might be some dimension mismatch. @weiyuan-jiang @tclune have you run into the issue? |
|
@junwang-noaa There should not be any halo inside of GOCART, so it is quite weird/interesting that your fix had any effect. |
|
Yes, this divide by 0 error is with intel debug The only change for the debug test is to shorten forecast length fhmax, since it is slower |
|
I didn’t run into this problem. Since this is in debug mode, weird thing may happen. In your case if what you pass in is zero size array ( like i2<i1, or j2<j1 or km =0), the loop is safe. But I am not sure about the vector form ( it should be fine, but who knows). Would you please check the size first ?
Cheers
Weiyuan
From: Jun Wang ***@***.***>
Date: Friday, September 20, 2024 at 1:43 PM
To: ufs-community/ufs-weather-model ***@***.***>
Cc: Jiang, Weiyuan (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] ***@***.***>, Mention ***@***.***>
Subject: [EXTERNAL] [BULK] Re: [ufs-community/ufs-weather-model] Add GEFS regression test suite from EP5r2 configuration/case (PR #2442)
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
@lipan-NOAA<https://github.com/lipan-NOAA> @junwang-noaa<https://github.com/junwang-noaa> Intel debug reliably fails in GOCART with traceback to Floating point exception: floating-point divide by zero at GOCART/Process_Library/GOCART2G_Process.F90:1575<https://github.com/GEOS-ESM/GOCART/blob/041422934cae1570f2f0e67239d5d89f11c6e1b7/Process_Library/GOCART2G_Process.F90#L1575> on n_atmsteps = 16: tau = vs/dz
I would imagine FV3 would complain first about any vanishing layer thickness (?), from say stochastic physics
Interestingly, a 3 hour forecast completes if I change this line to:
diff --git a/Process_Library/GOCART2G_Process.F90 b/Process_Library/GOCART2G_Process.F90
index cc0b599..123260c 100644
…--- a/Process_Library/GOCART2G_Process.F90
+++ b/Process_Library/GOCART2G_Process.F90
@@ -1561,7 +1561,7 @@ CONTAINS
! local
- integer :: i, j, iit
+ integer :: i, j, iit, k
integer :: nSubSteps
real, dimension(i1:i2, j1:j2, km) :: tau
@@ -1571,8 +1571,13 @@ CONTAINS
real :: dt, dt_cfl
-
- tau = vs/dz
+ do k = 1,km
+ do j = j1, j2
+ do i = i1, i2
+ tau(i,j,k) = vs(i,j,k)/dz(i,j,k)
+ end do
+ end do
+ end do
Maybe there is some haloes or padding to arrays leading to dz=0? I'm not sure where to go from here
@NickSzapiro-NOAA<https://github.com/NickSzapiro-NOAA> So you are running the test in debug mode? It might be some dimension mismatch.
@weiyuan-jiang<https://github.com/weiyuan-jiang> @tclune<https://github.com/tclune> have you run into the issue?
—
Reply to this email directly, view it on GitHub<#2442 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMQTYONHLCJBHPMPD6GI3STZXRNFLAVCNFSM6AAAAABOQJKYXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRUGIYTGMJWG4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@weiyuan-jiang @tclune I added There are dozens of tasks (out of 768) with small dz, with counts ranging from 1 up to 111. All have the same size The curiosities continue as using the alternate loop with |
| fltng_pnt | ||
| lossless | ||
| pos_pert_fcst | ||
| 12 |
There was a problem hiding this comment.
@NickSzapiro-NOAA It seems to me the UPP control files "postxconfig-NT-gefs.txt" and "postxconfig-NT-gefs_FH00.txt" haven't been updated to the new format. I wonder if any grib2 files (inline post results) have been successfully generated from your new RT.
There was a problem hiding this comment.
Thank you @WenMeng-NOAA . These postxconfig files are from the provided EP5r2 workflow and stopped working after UPP update in #2326.
Do you know how to reformat these?
As temporary fix, I'm using the gfs postxconfig instead. This choice happens in tests/fv3_conf/cpld_control_run.IN
#inline post
if [ $WRITE_DOPOST = .true. ]; then
cp ${PATHRT}/parm/post_itag_gfs itag
cp ${PATHRT}/parm/postxconfig-NT-gfs.txt postxconfig-NT.txt
cp ${PATHRT}/parm/postxconfig-NT-gfs_FH00.txt postxconfig-NT_FH00.txt
cp ${PATHRT}/parm/params_grib2_tbl_new params_grib2_tbl_new
if [[ ${BMIC} == .true. ]]; then
cp ${PATHRT}/parm/post_itag_gefs itag
#copied "gefs" postxconfig files not working afer UFS #2326
#cp ${PATHRT}/parm/postxconfig-NT-gefs.txt postxconfig-NT.txt
#cp ${PATHRT}/parm/postxconfig-NT-gefs_FH00.txt postxconfig-NT_FH00.txt
cp ${PATHRT}/parm/postxconfig-NT-gfs.txt postxconfig-NT.txt
cp ${PATHRT}/parm/postxconfig-NT-gfs_FH00.txt postxconfig-NT_FH00.txt
cp ${PATHRT}/parm/params_grib2_tbl_new params_grib2_tbl_new
else
cp ${PATHRT}/parm/post_itag_gfs itag
cp ${PATHRT}/parm/postxconfig-NT-gfs.txt postxconfig-NT.txt
cp ${PATHRT}/parm/postxconfig-NT-gfs_FH00.txt postxconfig-NT_FH00.txt
cp ${PATHRT}/parm/params_grib2_tbl_new params_grib2_tbl_new
fi
fi
There was a problem hiding this comment.
@NickSzapiro-NOAA @lipan-NOAA If you provide me the UPP control files in xml format, I can generate the text files in new format for you.
There was a problem hiding this comment.
@NickSzapiro-NOAA I have regenerated "postxconfig-NT-gefs.txt" with the xml file "postcntrl_gefs.xml" provided by @lipan-NOAA. Please let me know if you pick it up from Hera or other machines.
There was a problem hiding this comment.
@WenMeng-NOAA Can you make PR to this branch with file changes? If not, happy to bring in from Hera
There was a problem hiding this comment.
@NickSzapiro-NOAA A PR was just submitted to your branch.
There was a problem hiding this comment.
An example run directory using the updated postxconfig files is on hera at:
/scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/updateToEP5/uwm_gefs_upp/tests/run_dir/cpld_control_gefs_intel/
wgrib2 -v seems reasonable, but please let me know if we can verify contents are ok
There was a problem hiding this comment.
@NickSzapiro-NOAA Your test results look good to me, except for missing aerosol fields. I will provide you changes for generating these aerosol fields from the inline post.
There was a problem hiding this comment.
I re-ran and see aerosol fields (same run_dir). Please let me know if there is anything more to resolve
There was a problem hiding this comment.
@NickSzapiro-NOAA Your test results look good to me. @lipan-NOAA Can you also validate aerosol fields in grib2 files?
Update the inline post control files for gefs.
tests/parm/post_itag_gefs
Outdated
| MODELNAME='GFS' | ||
| / | ||
| &NAMPGB | ||
| KPO=50,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.4, |
There was a problem hiding this comment.
@NickSzapiro-NOAA To generate aerosol fields from the inline post, please add change as:
from
KPO=50,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.4,
into
KPO=50,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,40.,30.,20.,15.,10.,7.,5.,3.,2.,1.,0.4,nasa_on=.true.,
There was a problem hiding this comment.
@NickSzapiro-NOAA Also copy all optics_luts_*_nasa.dat files from UPP/fix/chem to your run directory.
@NickSzapiro-NOAA can you check if dz is zero on any of grid pionts since the error message is "Floating point exception: floating-point divide by zero", we may then need to track where the zero value dz comes from. Thanks |
Commit Queue Requirements:
Description:
This PR updates the cpld_bmark_p8 tests to a prototype GEFS test case of fully coupled s2swa+IAU+stochastics physics, with configuration and warm starts from restarts of EP5r2 ensemble member 1 for 2021-03-25 06Z.
The EP5r2 test case was kindly provided by @bingfu-NOAA via @junwang-noaa with aerosol input data and configurations from @lipan-NOAA.
A separate INPUTDATA_ROOT_BMIC is no longer needed and is removed.
This PR is in a draft mode subject to meeting basic reproducibility/quality checks. The following have been tested on Hera:
GNU debug fails with possible system error of [../../../../../opal/mca/btl/tcp/btl_tcp_endpoint.c:730:mca_btl_tcp_endpoint_start_connect] bind on local address (removed) failed: Address already in use (98)
Intel debug fails in GOCART Floating point exception: floating-point divide by zero (GOCART/Process_Library/GOCART2G_Process.F90:1575) on n_atmsteps = 16
Input data is currently in user space on hera and scripts need updating once filepaths are in shared space.
Commit Message:
Priority:
Git Tracking
UFSWM:
Sub component Pull Requests:
UFSWM Blocking Dependencies:
Changes
Regression Test Changes (Please commit test_changes.list):
Input data Changes:
Library Changes/Upgrades:
Testing Log: