Errors for making ICS and LBCS using HRRR for SRW

[htan2013]

Dear all,

I tried to run a case by using HRRR for SRW. When it comes to Making ICS and LBCS, they both showed errors:

For ICS, the error is:

OPEN VERTICAL COORD FILE:
/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/fix_am/global_
hyblev.l65.txt

• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT sphum
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rainwat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_wat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT liq_wat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT snowwat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT o3mr
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT graupel
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rain_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT water_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT liq_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT sgs_tke
• CALL FieldCreate FOR TARGET GRID TEMPERATURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rainwat
• CALL FieldCreate FOR TARGET GRID PRESSURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_wat
• CALL FieldCreate FOR TARGET GRID VERTICAL VELOCITY BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT snowwat
• CALL FieldCreate FOR TARGET GRID UNSTAGGERED WINDS BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT graupel
• CALL FieldCreate FOR TARGET TERRAIN.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_nc
• CALL FieldCreate FOR TARGET SURFACE PRESSURE BEFORE ADJUSTMENT.
• CALL FieldRegridStore FOR ATMOSPHERIC FIELDS.
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rain_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT water_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT liq_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT sgs_tke
• CALL FieldCreate FOR TARGET GRID TEMPERATURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID PRESSURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID VERTICAL VELOCITY BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID UNSTAGGERED WINDS BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET TERRAIN.
• CALL FieldCreate FOR TARGET SURFACE PRESSURE BEFORE ADJUSTMENT.
• CALL FieldRegridStore FOR ATMOSPHERIC FIELDS.

OPEN VERTICAL COORD FILE:
/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/fix_am/global_
hyblev.l65.txt

• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT sphum
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT liq_wat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT o3mr
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rainwat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_wat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT snowwat
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT graupel
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT rain_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT water_nc
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT liq_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT ice_aero
• CALL FieldCreate FOR TARGET GRID TRACER BEFORE ADJUSTMENT sgs_tke
• CALL FieldCreate FOR TARGET GRID TEMPERATURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID PRESSURE BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID VERTICAL VELOCITY BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET GRID UNSTAGGERED WINDS BEFORE ADJUSTMENT.
• CALL FieldCreate FOR TARGET TERRAIN.
• CALL FieldCreate FOR TARGET SURFACE PRESSURE BEFORE ADJUSTMENT.
• CALL FieldRegridStore FOR ATMOSPHERIC FIELDS.
• FATAL ERROR: IN FieldRegridStore
• IOSTAT IS: 513
• FATAL ERROR: IN FieldRegridStore
• IOSTAT IS: 513
• FATAL ERROR: IN FieldRegridStore
• IOSTAT IS: 513
• FATAL ERROR: IN FieldRegridStore
• IOSTAT IS: 513
  MPT ERROR: Rank 3(g:3) is aborting with error code 999.
  Process ID: 13661, Host: r3i7n26, Program: /glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/bin/chgres_cube
  MPT Version: HPE MPT 2.25 08/14/21 03:05:20

MPT: --------stack traceback-------
MPT ERROR: MPI_COMM_WORLD rank 1 has terminated without calling MPI_Finalize()
aborting job

ERROR:
From script: "exregional_make_ics.sh"
Full path to script: "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/regional_workflow/scripts/exregional_make_ics.sh"
Call to executable (exec_fp) to generate surface and initial conditions
(ICs) files for the FV3-LAM failed:
exec_fp = "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/bin/chgres_cube"
The external model from which the ICs files are to be generated is:
EXTRN_MDL_NAME_ICS = "HRRR"
The external model files that are inputs to the executable (exec_fp) are
located in the following directory:
extrn_mdl_staging_dir = "/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRRR/for_ICS"
Exiting with nonzero status.

ERROR:
From script: "JREGIONAL_MAKE_ICS"
Full path to script: "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/regional_workflow/jobs/JREGIONAL_MAKE_ICS"
Call to ex-script corresponding to J-job "JREGIONAL_MAKE_ICS" failed.
Exiting with nonzero status.

For LBCS, the error is:

/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRR
R/for_LBCS/hrrr.t12z.wrfprsf01.grib2

• OPEN AND READ INPUT DATA GRIB2 FILE:
  /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRR
  R/for_LBCS/hrrr.t12z.wrfprsf01.grib2
• OPEN AND READ INPUT DATA GRIB2 FILE:
  /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRR
  R/for_LBCS/hrrr.t12z.wrfprsf01.grib2
• CALL FieldCreate FOR TARGET GRID LANDMASK.
• CALL FieldCreate FOR TARGET GRID SEAMASK.
• CALL FieldCreate FOR TARGET GRID LATITUDE.
• CALL FieldCreate FOR TARGET GRID LATITUDE_S.
• CALL FieldCreate FOR TARGET GRID LATITUDE_W.
• CALL FieldCreate FOR TARGET GRID LONGITUDE.
• CALL FieldCreate FOR TARGET GRID LONGITUDE_S.
• CALL FieldCreate FOR TARGET GRID LONGITUDE_W.
• CALL FieldCreate FOR TARGET GRID TERRAIN.
• CALL FieldScatter FOR TARGET GRID LANDMASK. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID SEAMASK. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID SEAMASK. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE_S. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE_S. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE_W. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LONGITUDE_W. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE_S. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE_S. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE_W. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID LATITUDE_W. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID TERRAIN. TILE IS: 1
• CALL FieldScatter FOR TARGET GRID TERRAIN. TILE IS: 1
• OPEN AND READ INPUT DATA GRIB2 FILE:
  /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRR
  R/for_LBCS/hrrr.t12z.wrfprsf01.grib2
• OPEN AND READ INPUT DATA GRIB2 FILE:
  /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRR
  R/for_LBCS/hrrr.t12z.wrfprsf01.grib2
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CORNER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CORNER COORDINATES.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• INPUT DATA ON LAMBERT CONFORMAL GRID.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CENTER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL GridCreateNoPeriDim FOR INPUT GRID.
• CALL FieldCreate FOR INPUT GRID LATITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldCreate FOR INPUT GRID LONGITUDE.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• COMPUTE GRID CELL CORNER COORDINATES.
• CALL FieldScatter FOR INPUT GRID LONGITUDE.
  MPT: shepherd terminated: r7i5n10.ib0.cheyenne.ucar.edu - job aborting

ERROR:
From script: "exregional_make_lbcs.sh"
Full path to script: "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/regional_workflow/scripts/exregional_make_lbcs.sh"
Call to executable (exec_fp) to generate lateral boundary conditions (LBCs)
file for the FV3-LAM for forecast hour fhr failed:
exec_fp = "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/bin/chgres_cube"
fhr = "1"
The external model from which the LBCs files are to be generated is:
EXTRN_MDL_NAME_LBCS = "HRRR"
The external model files that are inputs to the executable (exec_fp) are
located in the following directory:
extrn_mdl_staging_dir = "/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/HRRR/for_LBCS"
Exiting with nonzero status.

ERROR:
From script: "JREGIONAL_MAKE_LBCS"
Full path to script: "/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/regional_workflow/jobs/JREGIONAL_MAKE_LBCS"
Call to ex-script corresponding to J-job "JREGIONAL_MAKE_LBCS" failed.
Exiting with nonzero status.

Make_ICS.log
Make_LBCS.log
config.sh .hrrr.txt

For previous steps, they all worked. I have also attached the Config.sh, Make_ICS and Make_LBCS.
I really appreciate any help you can provide.

Thanks,
Haochen

[ShawnCebulaNOAA]

Thanks for the question Haochen, we just let the team know and they'll take a look into this!

Shawn Cebula, EPIC End User Support

[htan2013]

Thanks!

Thanks for the question Haochen, we just let the team know and they'll take a look into this!

Shawn Cebula, EPIC End User Support

[natalie-perlin]

@htan2013
Haochen,

What are the steps you do when configuring and setting up the experiment? Could you please briefly list the steps, starting
from "git chlone ..." I'm looking into the issue you reported now.

[htan2013]

@htan2013 Haochen,

What are the steps you do when configuring and setting up the experiment? Could you please briefly list the steps, starting from "git chlone ..." I'm looking into the issue you reported now.

@natalie-perlin
I used the precompiled version in Cheyenne since the HPC-Stack is already installed on L1 systems.

The followings are environments that I put in bashrc for compilation or running:

export CMAKE_C_COMPILER=mpicc
export CMAKE_CXX_COMPILER=mpicxx
export CMAKE_Fortran_COMPILER=mpif90
export CMAKE_Platform=cheyenne.intel

Set UFS experiment directory
export EXPTDIR=/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR

HPC-Stack
module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack
module load hpc

UFS SRW Path
export SRW=/glade/scratch/htan2013/UFS_2.0/ufs-srweather-app

Conda Env for UFS
module use -a /glade/scratch/ketefian/TEST_mike_pr326/ufs-srweather-app/modulefiles
module load wflow_cheyenne
conda activate /glade/p/ral/jntp/UFS_SRW_app/conda/regional_workflow

I have tested the same case by using GFS data and it worked. When I switched to HRRR, it showed the error above.
Since I am running with a 3km resolution, I should use HRRR instead of GFS.

[natalie-perlin]

@htan2013 -
Haochen, thank you for the details. There are few things that seem to be off. First, there is no /glade/scratch/htan2013/UFS_2.0/ufs-srweather-app
directory but rather
/glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app

It looks like the checkout branch of github ufs-srweather-app in /glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/ is release/public-v2. This is an older release, and since then at least conda environment location (regional_workflow) has changed in the subsequent release/public-v2.1.0 .
But could you please check the environment variable SRW path first, and see if this resolves the issue

[htan2013]

@htan2013 - Haochen, thank you for the details. There are few things that seem to be off. First, there is no /glade/scratch/htan2013/UFS_2.0/ufs-srweather-app directory but rather /glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app

It looks like the checkout branch of github ufs-srweather-app in /glade/scratch/htan2013/UFS_2.0/UFS_2.0/ufs-srweather-app/ is release/public-v2. This is an older release, and since then at least conda environment location (regional_workflow) has changed in the subsequent release/public-v2.1.0 . But could you please check the environment variable SRW path first, and see if this resolves the issue

@natalie-perlin
Hi Natalie,

I changed the SRW accordingly, and the same error occurred.
Error file is in /glade/scratch/htan2013/UFS_2.0/Qsub, Make_ICS.log

[natalie-perlin]

@htan2013
Hi Haochen, thanks for letting me know location of the logs. From the errors in the log file, it looks that there is problem with regridding, and weight matrix does not contain all the needed data complaining " There exist destination points (e.g. id=288) which can't be mapped to any source cell" . Please see a log file /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/2020030612/INPUT/tmp_ICS/PET00.ESMF_LogFile , which says (lines 17-22):

20230303 150523.663 ERROR            PET00 Destination id=288 NOT found in weight matrix.
20230303 150523.663 ERROR            PET00 ESMCI_Mesh_Regrid_Glue.C:363 ESMCI_regrid_create() Arguments are incompatible  - - There exist destination points (e.g. id=288) which can't be mapped to any source cell 
20230303 150523.663 ERROR            PET00 ESMCI_Mesh_Regrid_Glue.C:586 ESMCI_regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20230303 150523.663 ERROR            PET00 ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20230303 150523.663 ERROR            PET00 ESMF_Regrid.F90:368 ESMF_RegridStore Arguments are incompatible  - Internal subroutine call returned Error
20230303 150523.663 ERROR            PET00 ESMF_FieldRegrid.F90:1202 ESMF_FieldRegridStoreNX Arguments are incompatible  - Internal subroutine call returned Error

Do you know if you may need to have weight matrix pre-calculated for the grid, and have the ideas about remapping errors?

[htan2013]

@natalie-perlin
Hi Natalie, thank you for helping me identify the error.
I am not familiar with this weight matrix and remapping errors. I just switched from GFS to HRRR.

As you suggested, I have switched from UFS2.0 to UFS2.1. I do notice that there are some changes for config.yaml. I tried my best to transfer my old config.sh to new config.yaml.

However, when I am testing UFS2.1 with GFS data, I met this error when I make ICS:

All executables will be submitted with command 'mpirun -np 12'.
/glade/scratch/htan2013/UFS_2.1/ufs-srweather-app/scripts/exregional_make_ics.sh: line 95: /glade/scratch/htan2013/UFS_2.1/expt_dirs/TEST_3km_GFS//FV3GFS/for_ICS/extrn_mdl_var_defns.sh: No such file or directory

FATAL ERROR:
ERROR:
From script: "JREGIONAL_MAKE_ICS"
Full path to script: "/glade/scratch/htan2013/UFS_2.1/ufs-srweather-app/jobs/JREGIONAL_MAKE_ICS"
Call to ex-script corresponding to J-job "JREGIONAL_MAKE_ICS" failed.
Exiting with nonzero status.

I am not sure what caused that double forward splash in "TEST_3km_GFS//FV3GFS/". But I assume it shouldn't be like that. The path for Make_ICS.log and Make_LBCS.log is in /glade/scratch/htan2013/UFS_2.1/Qsub. Can you please help me this also?

Many thanks,
Haochen

[natalie-perlin]

@htan2013

Hi Haochen, please see my notes below.

1. Regarding your tests using UFS_2.0. You seem to have a custom-defined grid, and lower-resolution GFS was easily interpolated into your 3-km grid. When you attempt to use 3-km HRRR for initialization, I assume it could create problems to remap to your custom grid, i.e., 3-km --> 3-km. Just to test the concept that this is indeed a culprit, you may test a custom grid with finer resolution, e.g. 1-km grid box. If the interpolation of the HRRR is done successfully to 1-km custom grid, then similar resolutions of source (HRRR) and destination (your domain) grids are the problem.

2. Please note, that custom grids were not supported in UFS SRW v2.0, and have not yet been supported in the recent SRW v2.1.0, but there are few remarks on creating new grids in v2.1.0 Documentation:
   https://ufs-srweather-app.readthedocs.io/en/release-public-v2.1.0/LAMGrids.html#creating-user-generated-grids. You may need to create a PREDEF_GRID_NAME variable to name you domain, and some other customizations. Please refer to the above link!

3. As I understand, you seem to be running the workflow using wrapper scripts, not using the workflow manager. Please see notes in Documentation Section 5.4.2 about using wrapper scripts, and in particular, about additional variables that need to be set, PDY and cyc.
   If you look in your log file "Run_Get_ICS.log" and "Run_Get_LBCS.log", you find the error complaining about "PDY: unbound variable". So your previous tasks, run_get_ics and run_get_lbcs did not complete successfully, so the subsequent ones, make_ics and make_lbcs, could not run.

I hope this helps,
Natalie.

[htan2013]

@natalie-perlin
Thank you so much for your explanation!

1. I have tested using 1km custom grid in UFS2.0. I am wondering whether SRW truly support 1km resolution since this post mentioned that SRW does not support resolution high than 3km.

The make_ics (/glade/scratch/htan2013/UFS_2.0/Qsub) in still has an error:

/glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/fix_lam/C10008_mosaic.halo4.nc

• READ GRID FILE NAMES
• OPEN GRID FILE:
  /glade/scratch/htan2013/UFS_2.0/expt_dirs/TEST_3km_20200306_HRRR/fix_lam/C10008_grid.tile7.halo.n

FATAL ERROR: opening grid file: No such file or directory
STOP.
MPT ERROR: Rank 0(g:0) is aborting with error code 999.
Process ID: 40103, Host: r13i5n21, Program: /glade/scratch/htan2013/UFS_2.0/ufs-srweather-app/bin/chgres_cube
MPT Version: HPE MPT 2.25 08/14/21 03:05:20

My fix_lam directory has C10008_grid.tile7.halo4.nc but I am not sure why program couldn't find it.
It tried to open the ".n" file instead of ".nc".

The PET files seem to be ok:

20230306 095715.880 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 095715.880 INFO PET00 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20230306 095715.880 INFO PET00 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20230306 095715.880 INFO PET00 !!! FOR PRODUCTION RUNS, USE: !!!
20230306 095715.880 INFO PET00 !!! ESMF_LOGKIND_Multi_On_Error !!!
20230306 095715.880 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 095715.880 INFO PET00 Running with ESMF Version : v8.3.0b09
20230306 095715.880 INFO PET00 ESMF library build date/time: "May 2 2022" "10:04:18"
20230306 095715.880 INFO PET00 ESMF library build location : /glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09
20230306 095715.880 INFO PET00 ESMF_COMM : mpt
20230306 095715.880 INFO PET00 ESMF_MOAB : enabled
20230306 095715.880 INFO PET00 ESMF_LAPACK : enabled
20230306 095715.880 INFO PET00 ESMF_NETCDF : enabled
20230306 095715.880 INFO PET00 ESMF_PNETCDF : disabled
20230306 095715.880 INFO PET00 ESMF_PIO : enabled
20230306 095715.880 INFO PET00 ESMF_YAMLCPP : enabled
20230306 105913.618 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 105913.618 INFO PET00 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20230306 105913.618 INFO PET00 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20230306 105913.618 INFO PET00 !!! FOR PRODUCTION RUNS, USE: !!!
20230306 105913.618 INFO PET00 !!! ESMF_LOGKIND_Multi_On_Error !!!
20230306 105913.618 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 105913.618 INFO PET00 Running with ESMF Version : v8.3.0b09
20230306 105913.618 INFO PET00 ESMF library build date/time: "May 2 2022" "10:04:18"
20230306 105913.618 INFO PET00 ESMF library build location : /glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09
20230306 105913.618 INFO PET00 ESMF_COMM : mpt
20230306 105913.619 INFO PET00 ESMF_MOAB : enabled
20230306 105913.619 INFO PET00 ESMF_LAPACK : enabled
20230306 105913.619 INFO PET00 ESMF_NETCDF : enabled
20230306 105913.619 INFO PET00 ESMF_PNETCDF : disabled
20230306 105913.619 INFO PET00 ESMF_PIO : enabled
20230306 105913.619 INFO PET00 ESMF_YAMLCPP : enabled
20230306 125816.083 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 125816.084 INFO PET00 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20230306 125816.084 INFO PET00 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20230306 125816.084 INFO PET00 !!! FOR PRODUCTION RUNS, USE: !!!
20230306 125816.084 INFO PET00 !!! ESMF_LOGKIND_Multi_On_Error !!!
20230306 125816.084 INFO PET00 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20230306 125816.084 INFO PET00 Running with ESMF Version : v8.3.0b09
20230306 125816.084 INFO PET00 ESMF library build date/time: "May 2 2022" "10:04:18"
20230306 125816.084 INFO PET00 ESMF library build location : /glade/work/jongkim/stacks/hash/hpc-stack-esmf/pkg/v8.3.0b09
20230306 125816.084 INFO PET00 ESMF_COMM : mpt
20230306 125816.085 INFO PET00 ESMF_MOAB : enabled
20230306 125816.085 INFO PET00 ESMF_LAPACK : enabled
20230306 125816.085 INFO PET00 ESMF_NETCDF : enabled
20230306 125816.085 INFO PET00 ESMF_PNETCDF : disabled
20230306 125816.085 INFO PET00 ESMF_PIO : enabled
20230306 125816.085 INFO PET00 ESMF_YAMLCPP : enabled

4. What's the difference between customizing the grid in config.yaml and creating new grids by the way in the user guide? i.e. changing valid_param_vals.yaml and predef_grid_params.yaml?

5. I have added the PDY and cyc before. It turned out that I have to define them in bashrc instead of simply "export PDY=20221005 && export cyc=00". However, for steps like make_grid, make_sfc_clmi, and make_orog, there is warning "/glade/scratch/htan2013/UFS_2.1/ufs-srweather-app/ush/bash_utils/print_msg.sh: line 67: BASH_SOURCE[1]: unbound variable". Get ICS and LBCS works but Make_ICS still shows a new error:

All executables will be submitted with command 'mpirun -np 12'.
/glade/scratch/htan2013/UFS_2.1/ufs-srweather-app/scripts/exregional_make_ics.sh: line 368: EXTRN_MDL_FNS[1]: unbound variable
FATAL ERROR:
ERROR:
From script: "JREGIONAL_MAKE_ICS"
Full path to script: "/glade/scratch/htan2013/UFS_2.1/ufs-srweather-app/jobs/JREGIONAL_MAKE_ICS"
Call to ex-script corresponding to J-job "JREGIONAL_MAKE_ICS" failed.
Exiting with nonzero status.

[natalie-perlin]

@htan2013
Haochen,

1. Trying to reproduce your error with the SRW v2.0

   I was able to set a similar experiment for 1.5-km grid to initialize from HRRR. The first few stages, make_grid, get_ics, get_lbcs were run as serial jobs and completed without issues . I have submitted the next one, make_ics as a batch job, using a script similar to your 4_Make_ICS.sh , except for my own paths; see experiment directory
   /glade/scratch/nperlin/SRW/srw-rel-v2/expt_dirs/TEST_1p5km_20200306_HRRR/.
   The job successfully started, but got terminated after hitting a requested 2:30 time limit, the error reported in a log file Make_ICS.log.
   So something is still off, and as you mention this post , there is no support for grids less than 3km.

In your case , a truncated filename could be due to the job termination on one of the processors that affected the rest, but it was not the main cause of the failure.

Re: 4. Due to multiple stages of the workflow, it would be best to follow the approach describe in the Documentation.

Re: 5. Still need to investigate

[natalie-perlin]

@htan2013

Haochen,

Re: 4. There are more variables for a new grid that need to be set up than those listed in config.sh. If they are not found in set_predef_grid_params.sh for that grid, default variables are used. It would be up to you to make sure the default variables fit your other job or task parameters and are consistent throughout the workflow.

Re: 5. FV3GFS input files that you use are in grib2 format. Your config.yaml files says they are in "nemsio" format.

Please ignore the warnings such as "BASH_SOURCE[1]: unbound variable", they are corrected in later code versions.

[htan2013]

@natalie-perlin
Thanks for your reply!
4. I will check them to make sure.
5. I used config_utils.py to transfer my previous config.sh to config.yaml. I do notice the format option changed from grib2 to nemsio. I can run GFS data without any issues but I figure HRRR might be a better ICBC.

Reply to your previous post:
I checked your Make_ICS.log. Seems like the ESMF is having an issue remapping HRRR data?

[natalie-perlin]

@htan2013 -

It would be more productive to keep diagnosing the issues custom grid using SRW v2.1 at this point. The setup similar to yours for HRRR as ICS and LBCS, experiment directory EXPTDIR=/glade/scratch/nperlin/SRW/srw-rel-v2.1/expt_dirs/HRRR_3km_TEST, fails in make_ics and make_lbcs again. The failure appears to be in ESMF: "FATAL ERROR: IN FieldRegridStore", line 1912 in $EXPTDIR/log/make_ics_2022100612.log, or line 1937 in make_lbcs_2022100612.log.
ESMF logs ($EXPTDIR/2022100612/tmp_ICS, $EXPTDIR/2022100612/tmp_LBCS) indicate that there is still a remapping issue, similar to what you observed with v2.0.
Before reaching out to ESMF team for support, could you test few things to gather more information:

• Make sure your grid with halo points is fully inside the HRRR domain;
• Try moving a center of the grid to a different location, and verify that problem persists;
• Build and run the code with esmf debug version for collecting more specifics on runtime errors. To use the esmf debug version, modify your $SRW/modulefiles/build_cheyenne_intel.lua to load it, and change line 35 to:
  load(pathJoin("esmf", os.getenv("esmf_ver") or "8.3.0b09-debug"))
  ...and also change line 8 in $SRW/modulefiles/srw_common.lua to
  load_any("esmf/8.3.0b09", "esmf/8.3.0b09-debug")

[natalie-perlin]

@htan2013
Some update: testing with esmf debug did not add more information, see recent test in
/glade/scratch/nperlin/SRW/srw-rel-v2.1/expt_dirs/HRRR_3km_debug

[natalie-perlin]

@htan2013
Hi Haochen,

The tasks make_ics and make_lbcs completed successfully when I moved the center of your custom grid 2 degrees to SW (arbitrary number, to ensure it fully inside the HRRR domain). Please see the EXPTDIR=/glade/scratch/nperlin/SRW/srw-rel-v2.1/expt_dirs/HRRR_3km_debug2/ and ./log directory for the reference.
Reconfiguring your domain (moving it) would solve the remapping issue.

[htan2013]

@htan2013 Hi Haochen,

The tasks make_ics and make_lbcs completed successfully when I moved the center of your custom grid 2 degrees to SW (arbitrary number, to ensure it fully inside the HRRR domain). Please see the EXPTDIR=/glade/scratch/nperlin/SRW/srw-rel-v2.1/expt_dirs/HRRR_3km_debug2/ and ./log directory for the reference. Reconfiguring your domain (moving it) would solve the remapping issue.

@natalie-perlin
Thank you so much for testing this!!
It seems like the write-component grid doesn't have to be within the native ESG grid but has to be within the ICBC grid. Thank you so much for identifying this error!!!

[pernak18]

@htan2013 @natalie-perlin thank you so much for diving into this. i'm having the same problem and have been following you all and your guidance in near real time.

one thing i came across today is the RRFS_CONUScompact_3km predefined grid. it seems to be in the predef_grid_params.yaml specifically for this purpose -- i.e., RRFS with HRRR IC/BC. i just used it in my run on hera and got past make_ics and make_lbcs and am now running the forecast.

could this be an alternative solution?

thanks again!

[natalie-perlin]

@pernak18 - the grid RRFS_CONUScompact_3km is a grid for a specific test case. While you could likely reuse the name in predef_grid_params.yaml to minimize the changes required, e.g., to valid_param_vals.yaml, it may create issues when you attempt to run test suites or WE2E tests. It could be a more sound approach to add your newly defined grid to both predef_grid_params.yaml and valid_param_vals.yaml and avoid any potential ambiguities.