Skip to content

Commit

Permalink
HAFSv1 related developments and moving-nesting code restructure (NCAR…
Browse files Browse the repository at this point in the history
…#610)

* Performance optimization of moving nest.

* update atmos_model and FV3GFS_io read performance when io_layout=1,1 and allow one to override data integrity checks in FMS restart logic

* Add the following HAFS ccpp physics suites (@ChunxiZhang-NOAA and @BinLiu-NOAA):
suite_FV3_HAFS_v0_thompson.xml
suite_FV3_HAFS_v0_thompson_nonsst.xml
suite_FV3_HAFS_v0_thompson_noahmp.xml
suite_FV3_HAFS_v0_thompson_noahmp_nonsst.xml

* Update submodule UPP to point its latest develop branch as of 05/18/2022.

* Update submodule upp, which has the fix for regional latlon grid crossing the prime meridian.

* Only call atmosphere_fill_nest_cpl at the cap driver time steps (coupling time
steps). This is to reduce the overhead introduced by downscaling the coupling
variables from FV3ATM parent to nest.

* Removed reference to unused variable parent_x.

* FV3-related typedefs changes for the Hurricane PBL options

* Update submodule ccpp/physics, which added the tc_pbl option in the GFS sa-TKE
EDMF PBL scheme for HAFS/hurricane modeling.

* Adding upoff as a namelist parameter

* Update submodule atmos_cubed_sphere, which has updated the time string
in internal tracker output (fort.602, phtcf file).

* Restructure moving nest code from atmos_cubed_sphere into FV3 directory.

* Rename HAFS_v0 CCPP physics suites to HAFS_V1.

* Added namelist flag fv_timers to enable detailed performance timings; defaults to false.

* Removed special CMake handling of moving nest files.  This causes a switch from 'fp-model source' as they were compiled in atmos_cubed_sphere, to 'fp-model consistent' aligned with the FV3atm tree.  Minor rounding differences are noted in forecast results.

* Update submodule ccpp/physics and update tc_pbl standard and long names.

* Removed ifdef MOVING_NEST, as files are included/excluded by cmake

Co-authored-by: William Ramstrom <[email protected]>
Co-authored-by: Rusty.Benson <[email protected]>
Co-authored-by: AndrewHazelton <[email protected]>
Co-authored-by: Biju Thomas <[email protected]>
  • Loading branch information
5 people committed Dec 15, 2022
1 parent 58134a1 commit 1d97e1f
Show file tree
Hide file tree
Showing 21 changed files with 10,382 additions and 11 deletions.
36 changes: 35 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,40 @@ if(NOT PARALLEL_NETCDF)
endif()

if(MOVING_NEST)
list(APPEND _fv3atm_defs_private MOVING_NEST)
list(APPEND _fv3atm_defs_private MOVING_NEST MOIST_CAPPA USE_COND)
if(DEBUG)
list(APPEND _fv3atm_defs_private DEBUG)
endif()
if(GFS_PHYS)
list(APPEND _fv3atm_defs_private GFS_PHYS)
endif()
if(GFS_TYPES)
list(APPEND _fv3atm_defs_private GFS_TYPES)
endif()
if(USE_GFSL63)
list(APPEND _fv3atm_defs_private USE_GFSL63)
endif()
if(INTERNAL_FILE_NML)
list(APPEND _fv3atm_defs_private INTERNAL_FILE_NML)
endif()
if(ENABLE_QUAD_PRECISION)
list(APPEND _fv3atm_defs_private ENABLE_QUAD_PRECISION)
endif()
if(32BIT)
list(APPEND _fv3atm_defs_private OVERLOAD_R4 OVERLOAD_R8)
endif()

list(APPEND moving_nest_srcs
moving_nest/bounding_box.F90
moving_nest/fv_tracker.F90
moving_nest/fv_moving_nest.F90
moving_nest/fv_moving_nest_main.F90
moving_nest/fv_moving_nest_physics.F90
moving_nest/fv_moving_nest_types.F90
moving_nest/fv_moving_nest_utils.F90
)
else()
list(APPEND moving_nest_srcs "")
endif()

add_library(fv3atm
Expand All @@ -57,6 +90,7 @@ add_library(fv3atm
io/module_fv3_io_def.F90
io/module_write_internal_state.F90
io/module_wrt_grid_comp.F90
${moving_nest_srcs}
${POST_SRC}
)

Expand Down
32 changes: 28 additions & 4 deletions atmos_model.F90
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,11 @@ module atmos_model_mod
block_data_combine_fractions

#ifdef MOVING_NEST
use fv_moving_nest_main_mod, only: update_moving_nest, dump_moving_nest
use fv_moving_nest_main_mod, only: update_moving_nest, dump_moving_nest
use fv_moving_nest_main_mod, only: nest_tracker_init
use fv_moving_nest_main_mod, only: moving_nest_end, nest_tracker_end
use fv_moving_nest_types_mod, only: fv_moving_nest_init
use fv_tracker_mod, only: check_is_moving_nest, execute_tracker
#endif
!-----------------------------------------------------------------------

Expand Down Expand Up @@ -132,6 +136,7 @@ module atmos_model_mod
logical :: nested ! true if there is a nest
logical :: moving_nest_parent ! true if this grid has a moving nest child
logical :: is_moving_nest ! true if this is a moving nest grid
logical :: isAtCapTime ! true if currTime is at the cap driverClock's currTime
integer :: ngrids !
integer :: mygrid !
integer :: mlon, mlat
Expand Down Expand Up @@ -296,7 +301,7 @@ subroutine update_atmos_radiation_physics (Atmos)
! receives coupled fields through the above assign_importdata step. Thus,
! an extra step is needed to fill the coupling variables in the nest,
! by downscaling the coupling variables from its parent.
if (Atmos%ngrids > 1) then
if (Atmos%isAtCapTime .and. Atmos%ngrids > 1) then
if (GFS_control%cplocn2atm .or. GFS_control%cplwav2atm) then
call atmosphere_fill_nest_cpl(Atm_block, GFS_control, GFS_data)
endif
Expand Down Expand Up @@ -540,6 +545,7 @@ subroutine atmos_model_init (Atmos, Time_init, Time, Time_step)

!---- set the atmospheric model time ------

Atmos % isAtCapTime = .false.
Atmos % Time_init = Time_init
Atmos % Time = Time
Atmos % Time_step = Time_step
Expand All @@ -552,14 +558,21 @@ subroutine atmos_model_init (Atmos, Time_init, Time, Time_step)
!---------- (need name of CCPP suite definition file from input.nml) ---------
call atmosphere_init (Atmos%Time_init, Atmos%Time, Atmos%Time_step,&
Atmos%grid, Atmos%area)

#ifdef MOVING_NEST
call fv_moving_nest_init(Atm, mygrid)
call nest_tracker_init()
#endif
!-----------------------------------------------------------------------
call atmosphere_resolution (nlon, nlat, global=.false.)
call atmosphere_resolution (mlon, mlat, global=.true.)
call atmosphere_domain (Atmos%domain, Atmos%domain_for_read, Atmos%layout, &
Atmos%regional, Atmos%nested, &
Atmos%moving_nest_parent, Atmos%is_moving_nest, &
Atmos%ngrids, Atmos%mygrid, Atmos%pelist)
Atmos%moving_nest_parent = .false.
Atmos%is_moving_nest = .false.
#ifdef MOVING_NEST
call check_is_moving_nest(Atm, Atmos%mygrid, Atmos%ngrids, Atmos%is_moving_nest, Atmos%moving_nest_parent)
#endif
call atmosphere_diag_axes (Atmos%axes)
call atmosphere_etalvls (Atmos%ak, Atmos%bk, flip=flip_vc)

Expand Down Expand Up @@ -929,6 +942,9 @@ subroutine update_atmos_model_state (Atmos, rc)
call mpp_clock_begin(fv3Clock)
call mpp_clock_begin(updClock)
call atmosphere_state_update (Atmos%Time, GFS_data, IAU_Data, Atm_block, flip_vc)
#ifdef MOVING_NEST
call execute_tracker(Atm, mygrid, Atmos%Time, Atmos%Time_step)
#endif
call mpp_clock_end(updClock)
call mpp_clock_end(fv3Clock)

Expand Down Expand Up @@ -1031,6 +1047,14 @@ subroutine atmos_model_end (Atmos)
!-----------------------------------------------------------------------
!---- termination routine for atmospheric model ----

#ifdef MOVING_NEST
! Call this before atmosphere_end(), because that deallocates Atm
if (Atmos%is_moving_nest) then
call moving_nest_end()
call nest_tracker_end()
endif
#endif

call atmosphere_end (Atmos % Time, Atmos%grid, restart_endfcst)

if(restart_endfcst) then
Expand Down
6 changes: 5 additions & 1 deletion ccpp/data/GFS_typedefs.F90
Original file line number Diff line number Diff line change
Expand Up @@ -1180,6 +1180,7 @@ module GFS_typedefs
real(kind=kind_phys) :: rlmx !< maximum allowed mixing length in boundary layer mass flux scheme
real(kind=kind_phys) :: elmx !< maximum allowed dissipation mixing length in boundary layer mass flux scheme
integer :: sfc_rlm !< choice of near surface mixing length in boundary layer mass flux scheme
integer :: tc_pbl !< control for TC applications in the PBL scheme

!--- parameters for canopy heat storage (CHS) parameterization
real(kind=kind_phys) :: h0facu !< CHS factor for sensible heat flux in unstable surface layer
Expand Down Expand Up @@ -3363,6 +3364,7 @@ subroutine control_initialize (Model, nlunit, fn_nml, me, master, &
real(kind=kind_phys) :: rlmx = 300. !< maximum allowed mixing length in boundary layer mass flux scheme
real(kind=kind_phys) :: elmx = 300. !< maximum allowed dissipation mixing length in boundary layer mass flux scheme
integer :: sfc_rlm = 0 !< choice of near surface mixing length in boundary layer mass flux scheme
integer :: tc_pbl = 0 !< control for TC applications in the PBL scheme

!--- parameters for canopy heat storage (CHS) parameterization
real(kind=kind_phys) :: h0facu = 0.25
Expand Down Expand Up @@ -3566,7 +3568,7 @@ subroutine control_initialize (Model, nlunit, fn_nml, me, master, &
thsfc_loc, &
! vertical diffusion
xkzm_m, xkzm_h, xkzm_s, xkzminv, moninq_fac, dspfac, &
bl_upfr, bl_dnfr, rlmx, elmx, sfc_rlm, &
bl_upfr, bl_dnfr, rlmx, elmx, sfc_rlm, tc_pbl, &
!--- canopy heat storage parameterization
h0facu, h0facs, &
!--- cellular automata
Expand Down Expand Up @@ -4380,6 +4382,7 @@ subroutine control_initialize (Model, nlunit, fn_nml, me, master, &
Model%rlmx = rlmx
Model%elmx = elmx
Model%sfc_rlm = sfc_rlm
Model%tc_pbl = tc_pbl

!--- canopy heat storage parametrization
Model%h0facu = h0facu
Expand Down Expand Up @@ -6067,6 +6070,7 @@ subroutine control_print(Model)
print *, ' rlmx : ', Model%rlmx
print *, ' elmx : ', Model%elmx
print *, ' sfc_rlm : ', Model%sfc_rlm
print *, ' tc_pbl : ', Model%tc_pbl
print *, ' '
print *, 'parameters for canopy heat storage parametrization'
print *, ' h0facu : ', Model%h0facu
Expand Down
6 changes: 6 additions & 0 deletions ccpp/data/GFS_typedefs.meta
Original file line number Diff line number Diff line change
Expand Up @@ -5001,6 +5001,12 @@
units = none
dimensions = ()
type = integer
[tc_pbl]
standard_name = control_for_TC_applications_in_the_PBL_scheme
long_name = control for TC applications in the PBL scheme
units = none
dimensions = ()
type = integer
[h0facu]
standard_name = multiplicative_tuning_parameter_for_reduced_surface_heat_fluxes_due_to_canopy_heat_storage
long_name = canopy heat storage factor for sensible heat flux in unstable surface layer
Expand Down
2 changes: 1 addition & 1 deletion ccpp/physics
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_HAFS_v0_gfdlmp_tedmf" version="1">
<suite name="FV3_HAFS_v1_gfdlmp_tedmf" version="1">
<!-- <init></init> -->
<group name="fast_physics">
<subcycle loop="1">
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_HAFS_v0_gfdlmp_tedmf_nonsst" version="1">
<suite name="FV3_HAFS_v1_gfdlmp_tedmf_nonsst" version="1">
<!-- <init></init> -->
<group name="fast_physics">
<subcycle loop="1">
Expand Down
91 changes: 91 additions & 0 deletions ccpp/suites/suite_FV3_HAFS_v1_thompson.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_HAFS_v1_thompson" version="1">
<!-- <init></init> -->
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>GFS_radiation_surface</scheme>
<scheme>rad_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_nst_pre</scheme>
<scheme>sfc_nst</scheme>
<scheme>sfc_nst_post</scheme>
<scheme>lsm_noah</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>satmedmfvdifq</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys_2015</scheme>
<scheme>h2ophys</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>mp_thompson_pre</scheme>
<scheme>mp_thompson</scheme>
<scheme>mp_thompson_post</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
<scheme>phys_tend</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
91 changes: 91 additions & 0 deletions ccpp/suites/suite_FV3_HAFS_v1_thompson_noahmp.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_HAFS_v1_thompson_noahmp" version="1">
<!-- <init></init> -->
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>GFS_radiation_surface</scheme>
<scheme>rad_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_nst_pre</scheme>
<scheme>sfc_nst</scheme>
<scheme>sfc_nst_post</scheme>
<scheme>noahmpdrv</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>satmedmfvdifq</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys_2015</scheme>
<scheme>h2ophys</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>mp_thompson_pre</scheme>
<scheme>mp_thompson</scheme>
<scheme>mp_thompson_post</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
<scheme>phys_tend</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
Loading

0 comments on commit 1d97e1f

Please sign in to comment.