HAFS fail at forecast task (Orion)

[t-connor-wx]

I am running HAFS on some research resolution GFS data and simulated observation data on Orion. It successfully gets through the analysis and analysis merge tasks, but fails at the forecast task. Investigation of the log file indicates that there seems to be a mismatch of PEs (not sure what those are). See error below:

• . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
  PROGRAM ufs HAS BEGUN. COMPILED 0.00 ORG: np23
  STARTING DATE-TIME OCT 04,2024 17:55:09.431 278 FRI 2460588

af ufs config,quilting= T calendar=julian iau_offset= 0 noutput_fh=
2
af ufs config,quilting= T write_groups= 3 20
isrcTermProcessing= 0
af ufs config,dt_atmos= 90 nfhmax= 126.0000
in fcst comp init, fcst_ntasks= 3000
af ufs config,num_restart_interval= 3
af ufs config,restart_interval= 3.000000 6.000000 9.000000
NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768.
&MPP_IO_NML
HEADER_BUFFER_VAL = 16384,
GLOBAL_FIELD_ON_ROOT_PE = T,
IO_CLOCKS_ON = F,
SHUFFLE = 0,
DEFLATE_LEVEL = -1,
CF_COMPLIANCE = F
/
NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 131072.
NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 30000000.
StartTime= 2019 10 13 0 0 0
CurrTime = 2019 10 13 0 0 0
StopTime = 2019 10 18 6 0 0
INPUT/coupler.res: date_init=2019 10 13 0 0 0
INPUT/coupler.res: date =2019 10 13 0 0 0
NOTE from PE 0: diag_manager_mod::diag_manager_init: diag_manager is using fms2_io
time_init= 2019 10 13 0 0
0 time= 2019 10 13 0 0
0 time_end= 2019 10 18 6
0 0 dt_atmos= 90
frestart= 3 6 9 0 0
0 0 0 0 0 total_inttime=
453600
atmosphere_init: current_time_seconds = 0.0
npes = 3000 , grid_pes = 1920 720

FATAL from PE 0: grid_pes in fv_nest_Nml does not assign all of the available PEs

FATAL from PE 0: grid_pes in fv_nest_Nml does not assign all of the available PEs

Looks like this is part of the UFS. Now, what it EXACTLY means….I am not sure. Looking at say fv_control.F90, this error is printed if pecounter /= npes and if mpp_pe() = 0. In that same script, pecounter is defined as:

do n=1,ngrids
.
.
.
pecounter = pecounter + 1
.
.
.

I don’t quite understand what this pecounter is supposed to be or what exactly this error implies? Any help here would be appreciated!

[jdkublnick]

Hello, t-connor-wx! Thanks for your patience; we have not forgotten about your question. We've contacted a subject matter expert on this topic for the most accurate response. As soon as we receive the information, we will follow up and provide you with an update.

[t-connor-wx]

Hello,

We have resolved the issue. In this specific run, I turned off the ocean and wav tasks and, thus, I needed to reset the PE counter to fix the issue in my rocoto script to include...

../parm/hfsa.conf forecast.all_tasks=2700