From aca835755aa82ead50040ea7e43ec63619667054 Mon Sep 17 00:00:00 2001 From: Tony Craig Date: Thu, 22 Feb 2024 08:55:44 -0800 Subject: [PATCH] Update IO formats and add new IO namelist controls (#928) This provides new features for CICE IO both thru netCDF and PIO. New namelist are added to control history and restart format, hdf5 compression and chunking, the PIO rearranger, and PIO IO task control. Separate controls are provided for history and restart files. The namelist changes are for _history_format, restart_format history_rearranger, restart_rearranger history_iotasks, history_root, history_stride, restart_iotasks, restart_root, and restart_stride history_chunksize, history_deflate, restart_chunksize, restart_deflate._ In particular, - Update restart_format and history_format options to 'cdf1', 'cdf2', 'cdf5', 'hdf5', 'pnetcdf1', 'pnetcdf2', 'pnetcdf5', 'default'. The old options, 'default', 'pio_netcdf', and 'pio_pnetcdf' are still supported and backwards compatible with lcdf64, but are deprecated and no longer documented. The old options and old namelist lcdf64 are covered by the new options. Support of the old options should be removed in the future. Note that some problems were discovered when opening files with hdf5 format but reading non-hdf5 files with a spack built PIO/netCDF. As a result, the format specified for the restart read is always 'cdf1' which provides flexibility and robustness across software installs, although it may result in serial reads of hdf5 files when a parallel read could be done. - Deprecate lcdf64 namelist. This namelist is no longer needed and is covered by the new restart_format and history_format options. The namelist still exists and is backwards compatible with the old 'default', 'pio_netcdf', and 'pio_pnetcdf' format options, but is no longer documented. This should be removed in the future. - Add new namelist to control PIO pe/task setup (iotasks, root, stride) for history and restart. These settings control the PIO IO tasks. The root, stride, and iotasks are consistent with the MPI communicator. root=0 is the first MPI task. These control PIO IO performance and are usually a function of things like the IO and node hardware. See PIO for more information. CICE computes PIO iotask, root, and stride defaults for cases where -99 is passed in for some or all of these namelist. Those defaults are somewhat constrained by a bug in PIO, https://github.com/NCAR/ParallelIO/issues/1986. The current implementation avoids the bug by limiting the iotasks for some MPI task counts. This is noted in ice_pio.F90. - Add new namelist to control PIO rearranger (rearranger) for history and restart. Supports 'box', 'subset', and 'default'. These control how PIO rearrangment is carried out. default is equivalent to box and the box generally performs better. See PIO for more information. - Add new namelist to support hdf5 compression and chunking (deflate, chunksize) for history and restart. The deflate controls file compression and is an integer between 0 and 9 where 0 means no compression and 9 is maximum compression. Generally, the higher the number, the slower the IO and the smaller the file, but the optimal setting depends on the contents of the file. Chunksize provides a performance control for the hdf5 parallel writes. It is a 2d array and is associated with the size of the piece of the array written by hdf5. hdf5 can be read and written in parallel, but that depends on how netCDF and PIO are built. Note that prior version of PIO, including PIO1, do not support the hdf5 compression and chunking thru the PIO interface. - Add new namelist settings (set_nml files) and update the io_suite to cover the new IO options. Remove old namelist settings associated with the deprecated format options and the lcdf64 namelist. These deprecated feature are no longer tested. - Update documentation to add new namelist and IO features. - Update the nuopc/cmeps driver code to support the new features. - Update the default ice_in to add the new namelist. - Update the derecho netcdf module to a version that supports hdf5. - Clean up some code formatting (indentation) --------- Co-authored-by: Anton Steketee --- .../cicedyn/analysis/ice_history_shared.F90 | 13 +- cicecore/cicedyn/general/ice_init.F90 | 303 ++++++++++++--- .../cicedyn/infrastructure/ice_read_write.F90 | 12 +- .../io/io_netcdf/ice_history_write.F90 | 347 ++++++++--------- .../io/io_netcdf/ice_restart.F90 | 52 ++- .../io/io_pio2/ice_history_write.F90 | 353 +++++++++--------- .../infrastructure/io/io_pio2/ice_pio.F90 | 156 +++++--- .../infrastructure/io/io_pio2/ice_restart.F90 | 76 +++- .../drivers/nuopc/cmeps/ice_comp_nuopc.F90 | 33 +- .../drivers/standalone/cice/CICE_InitMod.F90 | 2 +- cicecore/shared/ice_restart_shared.F90 | 11 +- configuration/scripts/ice_in | 12 + .../scripts/machines/env.derecho_cray | 2 + .../scripts/machines/env.derecho_gnu | 2 + .../scripts/machines/env.derecho_intel | 2 + .../scripts/machines/env.derecho_intelclassic | 2 + .../scripts/machines/env.derecho_inteloneapi | 2 + .../scripts/machines/env.derecho_nvhpc | 2 + configuration/scripts/options/set_env.iopio1p | 2 - configuration/scripts/options/set_env.iopio2p | 1 - .../scripts/options/set_nml.iobinary | 2 + configuration/scripts/options/set_nml.iocdf1 | 2 + configuration/scripts/options/set_nml.iocdf2 | 2 + configuration/scripts/options/set_nml.iocdf5 | 2 + configuration/scripts/options/set_nml.iohdf5 | 2 + .../scripts/options/set_nml.iohdf5opts | 4 + configuration/scripts/options/set_nml.iopio1 | 2 - configuration/scripts/options/set_nml.iopio1p | 2 - configuration/scripts/options/set_nml.iopio2 | 2 - configuration/scripts/options/set_nml.iopio2p | 2 - .../scripts/options/set_nml.iopioopts | 10 + .../scripts/options/set_nml.iopnetcdf1 | 2 + .../scripts/options/set_nml.iopnetcdf2 | 2 + .../scripts/options/set_nml.iopnetcdf5 | 2 + configuration/scripts/tests/io_suite.ts | 108 ++---- doc/source/cice_index.rst | 12 + doc/source/developer_guide/dg_infra.rst | 4 +- doc/source/user_guide/ug_case_settings.rst | 50 ++- doc/source/user_guide/ug_implementation.rst | 126 +++++-- 39 files changed, 1092 insertions(+), 631 deletions(-) delete mode 100644 configuration/scripts/options/set_env.iopio1p delete mode 100644 configuration/scripts/options/set_env.iopio2p create mode 100644 configuration/scripts/options/set_nml.iocdf1 create mode 100644 configuration/scripts/options/set_nml.iocdf2 create mode 100644 configuration/scripts/options/set_nml.iocdf5 create mode 100644 configuration/scripts/options/set_nml.iohdf5 create mode 100644 configuration/scripts/options/set_nml.iohdf5opts delete mode 100644 configuration/scripts/options/set_nml.iopio1 delete mode 100644 configuration/scripts/options/set_nml.iopio1p delete mode 100644 configuration/scripts/options/set_nml.iopio2 delete mode 100644 configuration/scripts/options/set_nml.iopio2p create mode 100644 configuration/scripts/options/set_nml.iopioopts create mode 100644 configuration/scripts/options/set_nml.iopnetcdf1 create mode 100644 configuration/scripts/options/set_nml.iopnetcdf2 create mode 100644 configuration/scripts/options/set_nml.iopnetcdf5 diff --git a/cicecore/cicedyn/analysis/ice_history_shared.F90 b/cicecore/cicedyn/analysis/ice_history_shared.F90 index 36f7f9131..ac2cf8afb 100644 --- a/cicecore/cicedyn/analysis/ice_history_shared.F90 +++ b/cicecore/cicedyn/analysis/ice_history_shared.F90 @@ -49,18 +49,23 @@ module ice_history_shared history_dir , & ! directory name for history file incond_dir ! directory for snapshot initial conditions - character (len=char_len_long), public :: & - pointer_file ! input pointer file for restarts - character (len=char_len), public :: & version_name character (len=char_len), public :: & - history_format + history_format , & ! history format, cdf1, cdf2, cdf5, etc + history_rearranger ! history file rearranger, box or subset for pio character (len=char_len), public :: & hist_suffix(max_nstrm) ! appended to 'h' in filename when not 'x' + integer (kind=int_kind), public :: & + history_iotasks , & ! iotasks, root, stride defines io pes for pio + history_root , & ! iotasks, root, stride defines io pes for pio + history_stride , & ! iotasks, root, stride defines io pes for pio + history_deflate , & ! compression level for hdf5/netcdf4 + history_chunksize(2) ! chunksize for hdf5/netcdf4 + !--------------------------------------------------------------- ! Instructions for adding a field: (search for 'example') ! Here or in ice_history_[process].F90: diff --git a/cicecore/cicedyn/general/ice_init.F90 b/cicecore/cicedyn/general/ice_init.F90 index 8875c7a29..24ac40db3 100644 --- a/cicecore/cicedyn/general/ice_init.F90 +++ b/cicecore/cicedyn/general/ice_init.F90 @@ -59,29 +59,37 @@ module ice_init subroutine input_data use ice_broadcast, only: broadcast_scalar, broadcast_array - use ice_diagnostics, only: diag_file, print_global, print_points, latpnt, lonpnt, & - debug_model, debug_model_step, debug_model_task, & - debug_model_i, debug_model_j, debug_model_iblk + use ice_diagnostics, only: & + diag_file, print_global, print_points, latpnt, lonpnt, & + debug_model, debug_model_step, debug_model_task, & + debug_model_i, debug_model_j, debug_model_iblk use ice_domain, only: close_boundaries, orca_halogrid - use ice_domain_size, only: ncat, nilyr, nslyr, nblyr, nfsd, nfreq, & - n_iso, n_aero, n_zaero, n_algae, & - n_doc, n_dic, n_don, n_fed, n_fep, & - max_nstrm - use ice_calendar, only: year_init, month_init, day_init, sec_init, & - istep0, histfreq, histfreq_n, histfreq_base, & - dumpfreq, dumpfreq_n, diagfreq, dumpfreq_base, & - npt, dt, ndtd, days_per_year, use_leap_years, & - write_ic, dump_last, npt_unit + use ice_domain_size, only: & + ncat, nilyr, nslyr, nblyr, nfsd, nfreq, & + n_iso, n_aero, n_zaero, n_algae, & + n_doc, n_dic, n_don, n_fed, n_fep, & + max_nstrm + use ice_calendar, only: & + year_init, month_init, day_init, sec_init, & + istep0, histfreq, histfreq_n, histfreq_base, & + dumpfreq, dumpfreq_n, diagfreq, dumpfreq_base, & + npt, dt, ndtd, days_per_year, use_leap_years, & + write_ic, dump_last, npt_unit use ice_arrays_column, only: oceanmixed_ice - use ice_restart_column, only: restart_age, restart_FY, restart_lvl, & + use ice_restart_column, only: & + restart_age, restart_FY, restart_lvl, & restart_pond_lvl, restart_pond_topo, restart_aero, & restart_fsd, restart_iso, restart_snow use ice_restart_shared, only: & - restart, restart_ext, restart_coszen, restart_dir, restart_file, pointer_file, & - runid, runtype, use_restart_time, restart_format, lcdf64 - use ice_history_shared, only: hist_avg, history_dir, history_file, hist_suffix, & - incond_dir, incond_file, version_name, & - history_precision, history_format, hist_time_axis + restart, restart_ext, restart_coszen, use_restart_time, & + runtype, restart_file, restart_dir, runid, pointer_file, & + restart_format, restart_rearranger, restart_iotasks, restart_root, & + restart_stride, restart_deflate, restart_chunksize + use ice_history_shared, only: & + history_precision, hist_avg, history_format, history_file, incond_file, & + history_dir, incond_dir, version_name, history_rearranger, & + hist_suffix, history_iotasks, history_root, history_stride, & + history_deflate, history_chunksize, hist_time_axis use ice_flux, only: update_ocn_f, cpl_frazil, l_mpond_fresh use ice_flux, only: default_season use ice_flux_bgc, only: cpl_bgc @@ -97,29 +105,31 @@ subroutine input_data snw_tau_fname, snw_kappa_fname, snw_drdt0_fname, & snw_rhos_fname, snw_Tgrd_fname, snw_T_fname use ice_arrays_column, only: bgc_data_dir, fe_data_type - use ice_grid, only: grid_file, gridcpl_file, kmt_file, & - bathymetry_file, use_bathymetry, & - bathymetry_format, kmt_type, & - grid_type, grid_format, & - grid_ice, grid_ice_thrm, grid_ice_dynu, grid_ice_dynv, & - grid_ocn, grid_ocn_thrm, grid_ocn_dynu, grid_ocn_dynv, & - grid_atm, grid_atm_thrm, grid_atm_dynu, grid_atm_dynv, & - dxrect, dyrect, dxscale, dyscale, scale_dxdy, & - lonrefrect, latrefrect, save_ghte_ghtn - use ice_dyn_shared, only: ndte, kdyn, revised_evp, yield_curve, & - evp_algorithm, visc_method, & - seabed_stress, seabed_stress_method, & - k1, k2, alphab, threshold_hw, Ktens, & - e_yieldcurve, e_plasticpot, coriolis, & - ssh_stress, kridge, brlx, arlx, & - deltaminEVP, deltaminVP, capping, & - elasticDamp - - use ice_dyn_vp, only: maxits_nonlin, precond, dim_fgmres, dim_pgmres, maxits_fgmres, & - maxits_pgmres, monitor_nonlin, monitor_fgmres, & - monitor_pgmres, reltol_nonlin, reltol_fgmres, reltol_pgmres, & - algo_nonlin, fpfunc_andacc, dim_andacc, reltol_andacc, & - damping_andacc, start_andacc, use_mean_vrel, ortho_type + use ice_grid, only: & + grid_file, gridcpl_file, kmt_file, & + bathymetry_file, use_bathymetry, & + bathymetry_format, kmt_type, & + grid_type, grid_format, & + grid_ice, grid_ice_thrm, grid_ice_dynu, grid_ice_dynv, & + grid_ocn, grid_ocn_thrm, grid_ocn_dynu, grid_ocn_dynv, & + grid_atm, grid_atm_thrm, grid_atm_dynu, grid_atm_dynv, & + dxrect, dyrect, dxscale, dyscale, scale_dxdy, & + lonrefrect, latrefrect, save_ghte_ghtn + use ice_dyn_shared, only: & + ndte, kdyn, revised_evp, yield_curve, & + evp_algorithm, visc_method, & + seabed_stress, seabed_stress_method, & + k1, k2, alphab, threshold_hw, Ktens, & + e_yieldcurve, e_plasticpot, coriolis, & + ssh_stress, kridge, brlx, arlx, & + deltaminEVP, deltaminVP, capping, & + elasticDamp + use ice_dyn_vp, only: & + maxits_nonlin, precond, dim_fgmres, dim_pgmres, maxits_fgmres, & + maxits_pgmres, monitor_nonlin, monitor_fgmres, & + monitor_pgmres, reltol_nonlin, reltol_fgmres, reltol_pgmres, & + algo_nonlin, fpfunc_andacc, dim_andacc, reltol_andacc, & + damping_andacc, start_andacc, use_mean_vrel, ortho_type use ice_transport_driver, only: advection, conserv_check use ice_restoring, only: restore_ice use ice_timers, only: timer_stats @@ -163,6 +173,7 @@ subroutine input_data logical (kind=log_kind) :: tr_iso, tr_aero, tr_fsd, tr_snow logical (kind=log_kind) :: tr_pond_lvl, tr_pond_topo integer (kind=int_kind) :: numin, numax ! unit number limits + logical (kind=log_kind) :: lcdf64 ! deprecated, backwards compatibility integer (kind=int_kind) :: rplvl, rptopo real (kind=dbl_kind) :: Cf, ksno, puny, ice_ref_salinity, Tocnfrz @@ -183,12 +194,15 @@ subroutine input_data runtype, runid, bfbflag, numax, & ice_ic, restart, restart_dir, restart_file, & restart_ext, use_restart_time, restart_format, lcdf64, & + restart_root, restart_stride, restart_iotasks, restart_rearranger, & + restart_deflate, restart_chunksize, & pointer_file, dumpfreq, dumpfreq_n, dump_last, & diagfreq, diag_type, diag_file, history_format,& + history_root, history_stride, history_iotasks, history_rearranger, & hist_time_axis, & print_global, print_points, latpnt, lonpnt, & debug_forcing, histfreq, histfreq_n, hist_avg, & - hist_suffix, & + hist_suffix, history_deflate, history_chunksize, & history_dir, history_file, history_precision, cpl_bgc, & histfreq_base, dumpfreq_base, timer_stats, memory_stats, & conserv_check, debug_model, debug_model_step, & @@ -326,20 +340,25 @@ subroutine input_data histfreq_base(:) = 'zero' ! output frequency reference date hist_avg(:) = .true. ! if true, write time-averages (not snapshots) hist_suffix(:) = 'x' ! appended to 'history_file' in filename when not 'x' - history_format = 'default' ! history file format + history_format = 'cdf1'! history file format + history_root = -99 ! history iotasks, root, stride sets pes for pio + history_stride = -99 ! history iotasks, root, stride sets pes for pio + history_iotasks = -99 ! history iotasks, root, stride sets pes for pio + history_rearranger = 'default' ! history rearranger for pio hist_time_axis = 'end' ! History file time axis averaging interval position - history_dir = './' ! write to executable dir for default history_file = 'iceh' ! history file name prefix history_precision = 4 ! precision of history files + history_deflate = 0 ! compression level for netcdf4 + history_chunksize(:) = 0 ! chunksize for netcdf4 write_ic = .false. ! write out initial condition cpl_bgc = .false. ! couple bgc thru driver incond_dir = history_dir ! write to history dir for default incond_file = 'iceh_ic'! file prefix - dumpfreq(:)='x' ! restart frequency option + dumpfreq(:) = 'x' ! restart frequency option dumpfreq_n(:) = 1 ! restart frequency dumpfreq_base(:) = 'init' ! restart frequency reference date - dumpfreq(1)='y' ! restart frequency option + dumpfreq(1) = 'y' ! restart frequency option dumpfreq_n(1) = 1 ! restart frequency dump_last = .false. ! write restart on last time step restart_dir = './' ! write to executable dir for default @@ -347,7 +366,13 @@ subroutine input_data restart_ext = .false. ! if true, read/write ghost cells restart_coszen = .false. ! if true, read/write coszen pointer_file = 'ice.restart_file' - restart_format = 'default' ! restart file format + restart_format = 'cdf1' ! restart file format + restart_root = -99 ! restart iotasks, root, stride sets pes for pio + restart_stride = -99 ! restart iotasks, root, stride sets pes for pio + restart_iotasks = -99 ! restart iotasks, root, stride sets pes for pio + restart_rearranger = 'default' ! restart rearranger for pio + restart_deflate = 0 ! compression level for netcdf4 + restart_chunksize(:) = 0 ! chunksize for netcdf4 lcdf64 = .false. ! 64 bit offset for netCDF ice_ic = 'default' ! latitude and sst-dependent grid_format = 'bin' ! file format ('bin'=binary or 'nc'=netcdf) @@ -922,7 +947,13 @@ subroutine input_data call broadcast_scalar(history_file, master_task) call broadcast_scalar(history_precision, master_task) call broadcast_scalar(history_format, master_task) + call broadcast_scalar(history_iotasks, master_task) + call broadcast_scalar(history_root, master_task) + call broadcast_scalar(history_stride, master_task) + call broadcast_scalar(history_rearranger, master_task) call broadcast_scalar(hist_time_axis, master_task) + call broadcast_scalar(history_deflate, master_task) + call broadcast_array(history_chunksize, master_task) call broadcast_scalar(write_ic, master_task) call broadcast_scalar(cpl_bgc, master_task) call broadcast_scalar(incond_dir, master_task) @@ -935,6 +966,12 @@ subroutine input_data call broadcast_scalar(restart_coszen, master_task) call broadcast_scalar(use_restart_time, master_task) call broadcast_scalar(restart_format, master_task) + call broadcast_scalar(restart_iotasks, master_task) + call broadcast_scalar(restart_root, master_task) + call broadcast_scalar(restart_stride, master_task) + call broadcast_scalar(restart_rearranger, master_task) + call broadcast_scalar(restart_deflate, master_task) + call broadcast_array(restart_chunksize, master_task) call broadcast_scalar(lcdf64, master_task) call broadcast_scalar(pointer_file, master_task) call broadcast_scalar(ice_ic, master_task) @@ -1232,6 +1269,95 @@ subroutine input_data abort_list = trim(abort_list)//":1" endif + if (history_format /= 'cdf1' .and. & + history_format /= 'cdf2' .and. & + history_format /= 'cdf5' .and. & + history_format /= 'hdf5' .and. & + history_format /= 'pnetcdf1' .and. & + history_format /= 'pnetcdf2' .and. & + history_format /= 'pnetcdf5' .and. & + history_format /= 'pio_netcdf' .and. & ! backwards compatibility + history_format /= 'pio_pnetcdf' .and. & ! backwards compatibility + history_format /= 'binary' .and. & + history_format /= 'default') then ! backwards compatibility + if (my_task == master_task) then + write(nu_diag,*) subname//' ERROR: history_format unknown = ',trim(history_format) + endif + abort_list = trim(abort_list)//":50" + endif + + if (restart_format /= 'cdf1' .and. & + restart_format /= 'cdf2' .and. & + restart_format /= 'cdf5' .and. & + restart_format /= 'hdf5' .and. & + restart_format /= 'pnetcdf1' .and. & + restart_format /= 'pnetcdf2' .and. & + restart_format /= 'pnetcdf5' .and. & + restart_format /= 'pio_netcdf' .and. & ! backwards compatibility + restart_format /= 'pio_pnetcdf' .and. & ! backwards compatibility + restart_format /= 'binary' .and. & + restart_format /= 'default') then ! backwards compatibility + if (my_task == master_task) then + write(nu_diag,*) subname//' ERROR: restart_format unknown = ',trim(restart_format) + endif + abort_list = trim(abort_list)//":51" + endif + + ! backwards compatibility for history and restart formats, lcdf64 + + if (history_format == 'pio_pnetcdf' .or. history_format == 'pio_netcdf') then + if (my_task == master_task) then + write(nu_diag,*) subname//' WARNING: history_format='//trim(history_format)// & + ' is deprecated, please update namelist settings' + endif + endif + if (restart_format == 'pio_pnetcdf' .or. restart_format == 'pio_netcdf') then + if (my_task == master_task) then + write(nu_diag,*) subname//' WARNING: restart_format='//trim(restart_format)// & + ' is deprecated, please update namelist settings' + endif + endif + + if (lcdf64) then + if (my_task == master_task) then + write(nu_diag,*) subname//' WARNING: lcdf64 is deprecated, please update namelist settings' + endif + + if (history_format == 'default' .or. history_format == 'pio_netcdf') then + history_format = 'cdf2' + elseif (history_format == 'pio_pnetcdf') then + history_format = 'pnetcdf2' + else + if (my_task == master_task) then + write(nu_diag,*) subname//' ERROR: lcdf64 is T and history_format not supported for '//trim(history_format) + endif + abort_list = trim(abort_list)//":52" + endif + + if (restart_format == 'default' .or. restart_format == 'pio_netcdf') then + restart_format = 'cdf2' + elseif (restart_format == 'pio_pnetcdf') then + restart_format = 'pnetcdf2' + else + if (my_task == master_task) then + write(nu_diag,*) subname//' ERROR: lcdf64 is T and restart_format not supported for '//trim(restart_format) + endif + abort_list = trim(abort_list)//":53" + endif + else + if (history_format == 'default' .or. history_format == 'pio_netcdf') then + history_format = 'cdf1' + elseif (history_format == 'pio_pnetcdf') then + history_format = 'pnetcdf1' + endif + + if (restart_format == 'default' .or. restart_format == 'pio_netcdf') then + restart_format = 'cdf1' + elseif (restart_format == 'pio_pnetcdf') then + restart_format = 'pnetcdf1' + endif + endif + if (ktransport <= 0) then advection = 'none' endif @@ -1504,7 +1630,7 @@ subroutine input_data write (nu_diag,*) subname//' ERROR: snow grain radius is activated' write (nu_diag,*) subname//' ERROR: Must use shortwave=dEdd or dEdd_snicar_ad' endif - abort_list = trim(abort_list)//":29" + abort_list = trim(abort_list)//":17" endif if ((rfracmin < -puny .or. rfracmin > c1+puny) .or. & @@ -1590,18 +1716,18 @@ subroutine input_data abort_list = trim(abort_list)//":19" endif - if(history_precision .ne. 4 .and. history_precision .ne. 8) then + if (history_precision .ne. 4 .and. history_precision .ne. 8) then write (nu_diag,*) subname//' ERROR: bad value for history_precision, allowed values: 4, 8' abort_list = trim(abort_list)//":22" endif do n = 1,max_nstrm - if(histfreq_base(n) /= 'init' .and. histfreq_base(n) /= 'zero') then + if (histfreq_base(n) /= 'init' .and. histfreq_base(n) /= 'zero') then write (nu_diag,*) subname//' ERROR: bad value for histfreq_base, allowed values: init, zero: '//trim(histfreq_base(n)) abort_list = trim(abort_list)//":24" endif - if(dumpfreq_base(n) /= 'init' .and. dumpfreq_base(n) /= 'zero') then + if (dumpfreq_base(n) /= 'init' .and. dumpfreq_base(n) /= 'zero') then write (nu_diag,*) subname//' ERROR: bad value for dumpfreq_base, allowed values: init, zero: '//trim(dumpfreq_base(n)) abort_list = trim(abort_list)//":25" endif @@ -1616,11 +1742,63 @@ subroutine input_data endif enddo - if(trim(hist_time_axis) /= 'begin' .and. trim(hist_time_axis) /= 'middle' .and. trim(hist_time_axis) /= 'end') then + if (trim(hist_time_axis) /= 'begin' .and. trim(hist_time_axis) /= 'middle' .and. trim(hist_time_axis) /= 'end') then write (nu_diag,*) subname//' ERROR: hist_time_axis value not valid = '//trim(hist_time_axis) abort_list = trim(abort_list)//":29" endif +#ifdef USE_PIO1 + if (history_deflate/=0 .or. restart_deflate/=0 .or. & + history_chunksize(1)/=0 .or. history_chunksize(2)/=0 .or. & + restart_chunksize(1)/=0 .or. restart_chunksize(2)/=0) then + if (my_task == master_task) write (nu_diag,*) subname//' ERROR: _deflate and _chunksize not compatible with PIO1' + abort_list = trim(abort_list)//":54" + endif +#else +#ifndef CESMCOUPLED + ! history_format not used by nuopc driver + if (history_format/='hdf5' .and. history_deflate/=0) then + if (my_task == master_task) then + write (nu_diag,*) subname//' WARNING: history_deflate not compatible with '//history_format + write (nu_diag,*) subname//' WARNING: netcdf compression only possible with history_type="hdf5" ' + endif + endif + + if (history_format/='hdf5' .and. (history_chunksize(1)/=0 .or. history_chunksize(2)/=0)) then + if (my_task == master_task) then + write (nu_diag,*) subname//' WARNING: history_chunksize not compatible with '//history_format + write (nu_diag,*) subname//' WARNING: netcdf chunking only possible with history_type="hdf5" ' + endif + endif + + if (restart_format/='hdf5' .and. restart_deflate/=0) then + if (my_task == master_task) then + write (nu_diag,*) subname//' WARNING: restart_deflate not compatible with '//restart_format + write (nu_diag,*) subname//' WARNING: netcdf compression only possible with restart_type="hdf5" ' + endif + endif + + if (restart_format/='hdf5' .and. (restart_chunksize(1)/=0 .or. restart_chunksize(2)/=0)) then + if (my_task == master_task) then + write (nu_diag,*) subname//' WARNING: restart_chunksize not compatible with '//restart_format + write (nu_diag,*) subname//' WARNING: netcdf chunking only possible with restart_type="hdf5" ' + endif + endif +#endif + + if (history_deflate<0 .or. history_deflate>9) then + if (my_task == master_task) write (nu_diag,*) subname//& + ' ERROR: history_deflate value not valid. Allowed range: integers from 0 to 9 ' + abort_list = trim(abort_list)//":55" + endif + + if (restart_deflate<0 .or. restart_deflate>9) then + if (my_task == master_task) write (nu_diag,*) subname//& + ' ERROR: restart_deflate value not valid. Allowed range: integers from 0 to 9 ' + abort_list = trim(abort_list)//":56" + endif +#endif + ! Implicit solver input validation if (kdyn == 3) then if (.not. (trim(algo_nonlin) == 'picard' .or. trim(algo_nonlin) == 'anderson')) then @@ -2164,7 +2342,7 @@ subroutine input_data tmpstr2 = ' : dragio hard-coded' endif write(nu_diag,1010) ' calc_dragio = ', calc_dragio,trim(tmpstr2) - if(calc_dragio) then + if (calc_dragio) then write(nu_diag,1002) ' iceruf_ocn = ', iceruf_ocn,' : under-ice roughness length' endif @@ -2357,13 +2535,19 @@ subroutine input_data write(nu_diag,1033) ' histfreq = ', histfreq(:) write(nu_diag,1023) ' histfreq_n = ', histfreq_n(:) write(nu_diag,1033) ' histfreq_base = ', histfreq_base(:) - write(nu_diag,*) ' hist_avg = ', hist_avg(:) + write(nu_diag,1013) ' hist_avg = ', hist_avg(:) write(nu_diag,1033) ' hist_suffix = ', hist_suffix(:) write(nu_diag,1031) ' history_dir = ', trim(history_dir) write(nu_diag,1031) ' history_file = ', trim(history_file) write(nu_diag,1021) ' history_precision= ', history_precision write(nu_diag,1031) ' history_format = ', trim(history_format) + write(nu_diag,1031) ' history_rearranger = ', trim(history_rearranger) + write(nu_diag,1021) ' history_iotasks = ', history_iotasks + write(nu_diag,1021) ' history_root = ', history_root + write(nu_diag,1021) ' history_stride = ', history_stride write(nu_diag,1031) ' hist_time_axis = ', trim(hist_time_axis) + write(nu_diag,1021) ' history_deflate = ', history_deflate + write(nu_diag,1023) ' history_chunksize= ', history_chunksize if (write_ic) then write(nu_diag,1039) ' Initial condition will be written in ', & trim(incond_dir) @@ -2377,7 +2561,13 @@ subroutine input_data write(nu_diag,1011) ' restart_ext = ', restart_ext write(nu_diag,1011) ' restart_coszen = ', restart_coszen write(nu_diag,1031) ' restart_format = ', trim(restart_format) - write(nu_diag,1011) ' lcdf64 = ', lcdf64 + write(nu_diag,1021) ' restart_deflate = ', restart_deflate + write(nu_diag,1023) ' restart_chunksize= ', restart_chunksize +! write(nu_diag,1011) ' lcdf64 = ', lcdf64 ! deprecated + write(nu_diag,1031) ' restart_rearranger = ', trim(restart_rearranger) + write(nu_diag,1021) ' restart_iotasks = ', restart_iotasks + write(nu_diag,1021) ' restart_root = ', restart_root + write(nu_diag,1021) ' restart_stride = ', restart_stride write(nu_diag,1031) ' restart_file = ', trim(restart_file) write(nu_diag,1031) ' pointer_file = ', trim(pointer_file) write(nu_diag,1011) ' use_restart_time = ', use_restart_time @@ -2402,7 +2592,7 @@ subroutine input_data if (trim(atm_data_type) /= 'default') then write(nu_diag,1031) ' atm_data_dir = ', trim(atm_data_dir) write(nu_diag,1031) ' precip_units = ', trim(precip_units) - elseif (trim(atm_data_type)=='default') then + elseif (trim(atm_data_type) == 'default') then write(nu_diag,1031) ' default_season = ', trim(default_season) endif @@ -2560,6 +2750,7 @@ subroutine input_data 1009 format (a20,1x,d13.6,1x,a) 1010 format (a20,8x,l6,1x,a) ! logical 1011 format (a20,1x,l6) + 1013 format (a20,1x,6l3) 1020 format (a20,8x,i6,1x,a) ! integer 1021 format (a20,1x,i6) 1022 format (a20,1x,i12) diff --git a/cicecore/cicedyn/infrastructure/ice_read_write.F90 b/cicecore/cicedyn/infrastructure/ice_read_write.F90 index ad50b38f2..4613843b5 100644 --- a/cicecore/cicedyn/infrastructure/ice_read_write.F90 +++ b/cicecore/cicedyn/infrastructure/ice_read_write.F90 @@ -1241,7 +1241,7 @@ subroutine ice_read_nc_xy(fid, nrec, varname, work, diag, & status = nf90_get_att(fid, varid, "_FillValue", missingvalue) ! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', & ! file=__FILE__, line=__LINE__) - write(nu_diag,*) subname,' missingvalue= ',missingvalue +! write(nu_diag,*) subname,' missingvalue= ',missingvalue amin = minval(work_g1) amax = maxval(work_g1, mask = work_g1 /= missingvalue) asum = sum (work_g1, mask = work_g1 /= missingvalue) @@ -1442,7 +1442,7 @@ subroutine ice_read_nc_xyz(fid, nrec, varname, work, diag, & status = nf90_get_att(fid, varid, "_FillValue", missingvalue) ! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', & ! file=__FILE__, line=__LINE__) - write(nu_diag,*) subname,' missingvalue= ',missingvalue +! write(nu_diag,*) subname,' missingvalue= ',missingvalue do n=1,ncat amin = minval(work_g1(:,:,n)) amax = maxval(work_g1(:,:,n), mask = work_g1(:,:,n) /= missingvalue) @@ -1654,7 +1654,7 @@ subroutine ice_read_nc_xyf(fid, nrec, varname, work, diag, & status = nf90_get_att(fid, varid, "_FillValue", missingvalue) ! call ice_check_nc(status, subname//' ERROR: Missing _FillValue', & ! file=__FILE__, line=__LINE__) - write(nu_diag,*) subname,' missingvalue= ',missingvalue +! write(nu_diag,*) subname,' missingvalue= ',missingvalue do n = 1, nfreq amin = minval(work_g1(:,:,n)) amax = maxval(work_g1(:,:,n), mask = work_g1(:,:,n) /= missingvalue) @@ -2589,13 +2589,13 @@ subroutine ice_check_nc(status, abort_msg, file, line) #ifdef USE_NETCDF if (status /= nf90_noerr) then if (present(file) .and. present(line)) then - call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg), & + call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg), & file=file, line=line) elseif (present(file)) then - call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg), & + call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg), & file=file) else - call abort_ice(subname//trim(nf90_strerror(status))//', '//trim(abort_msg)) + call abort_ice(subname//' '//trim(nf90_strerror(status))//', '//trim(abort_msg)) endif endif #else diff --git a/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90 b/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90 index a0e0ad3c2..c03bc233a 100644 --- a/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90 +++ b/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90 @@ -27,11 +27,29 @@ module ice_history_write use ice_read_write, only: ice_check_nc use icepack_intfc, only: icepack_warnings_flush, icepack_warnings_aborted use icepack_intfc, only: icepack_query_parameters + use ice_kinds_mod, only: int_kind +#ifdef USE_NETCDF + use netcdf +#endif implicit none private + + TYPE coord_attributes ! netcdf coordinate attributes + character (len=11) :: short_name + character (len=45) :: long_name + character (len=30) :: units + END TYPE coord_attributes + + TYPE req_attributes ! req'd netcdf attributes + type (coord_attributes) :: req + character (len=20) :: coordinates + END TYPE req_attributes + public :: ice_write_hist + integer (kind=int_kind) :: imtid,jmtid + !======================================================================= contains @@ -61,13 +79,9 @@ subroutine ice_write_hist (ns) lont_bounds, latt_bounds, lonu_bounds, latu_bounds, & lonn_bounds, latn_bounds, lone_bounds, late_bounds use ice_history_shared - use ice_restart_shared, only: lcdf64 #ifdef CESMCOUPLED use ice_restart_shared, only: runid #endif -#ifdef USE_NETCDF - use netcdf -#endif integer (kind=int_kind), intent(in) :: ns @@ -78,7 +92,7 @@ subroutine ice_write_hist (ns) real (kind=dbl_kind), dimension(nx_block,ny_block,max_blocks) :: work1 integer (kind=int_kind) :: i,k,ic,n,nn, & - ncid,status,imtid,jmtid,kmtidi,kmtids,kmtidb, cmtid,timid,varid, & + ncid,status,kmtidi,kmtids,kmtidb, cmtid,timid,varid, & nvertexid,ivertex,kmtida,iflag, fmtid integer (kind=int_kind), dimension(3) :: dimid integer (kind=int_kind), dimension(4) :: dimidz @@ -86,18 +100,19 @@ subroutine ice_write_hist (ns) integer (kind=int_kind), dimension(3) :: dimid_nverts integer (kind=int_kind), dimension(6) :: dimidex real (kind=dbl_kind) :: ltime2 - character (char_len) :: title + character (char_len) :: title, cal_units, cal_att character (char_len) :: time_period_freq = 'none' character (char_len_long) :: ncfile(max_nstrm) real (kind=dbl_kind) :: secday, rad_to_deg - integer (kind=int_kind) :: ind,boundid - - integer (kind=int_kind) :: lprecision + integer (kind=int_kind) :: ind,boundid, lprecision character (char_len) :: start_time,current_date,current_time character (len=8) :: cdate + ! time coord + TYPE(coord_attributes) :: time_coord + ! 8 coordinate variables: TLON, TLAT, ULON, ULAT, NLON, NLAT, ELON, ELAT INTEGER (kind=int_kind), PARAMETER :: ncoord = 8 @@ -108,17 +123,6 @@ subroutine ice_write_hist (ns) ! lont_bounds, latt_bounds, lonu_bounds, latu_bounds INTEGER (kind=int_kind), PARAMETER :: nvar_verts = 8 - TYPE coord_attributes ! netcdf coordinate attributes - character (len=11) :: short_name - character (len=45) :: long_name - character (len=20) :: units - END TYPE coord_attributes - - TYPE req_attributes ! req'd netcdf attributes - type (coord_attributes) :: req - character (len=20) :: coordinates - END TYPE req_attributes - TYPE(req_attributes), dimension(nvar_grd) :: var_grd TYPE(coord_attributes), dimension(ncoord) :: var_coord TYPE(coord_attributes), dimension(nvar_verts) :: var_nverts @@ -148,8 +152,18 @@ subroutine ice_write_hist (ns) endif ! create file - iflag = nf90_clobber - if (lcdf64) iflag = ior(iflag,nf90_64bit_offset) + if (history_format == 'cdf1') then + iflag = nf90_clobber + elseif (history_format == 'cdf2') then + iflag = ior(nf90_clobber,nf90_64bit_offset) + elseif (history_format == 'cdf5') then + iflag = ior(nf90_clobber,nf90_64bit_data) + elseif (history_format == 'hdf5') then + iflag = ior(nf90_clobber,nf90_netcdf4) + else + call abort_ice(subname//' ERROR: history_format not allowed for '//trim(history_format), & + file=__FILE__, line=__LINE__) + endif status = nf90_create(ncfile(ns), iflag, ncid) call ice_check_nc(status, subname// ' ERROR: creating history ncfile '//ncfile(ns), & file=__FILE__, line=__LINE__) @@ -205,83 +219,44 @@ subroutine ice_write_hist (ns) file=__FILE__, line=__LINE__) !----------------------------------------------------------------- - ! define coordinate variables + ! define coordinate variables: time, time_bounds !----------------------------------------------------------------- - status = nf90_def_var(ncid,'time',nf90_double,timid,varid) - call ice_check_nc(status, subname// ' ERROR: defining var time', & - file=__FILE__, line=__LINE__) - - status = nf90_put_att(ncid,varid,'long_name','time') - call ice_check_nc(status, subname// ' ERROR: time long_name', & - file=__FILE__, line=__LINE__) - write(cdate,'(i8.8)') idate0 - write(title,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & + write(cal_units,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & cdate(1:4),'-',cdate(5:6),'-',cdate(7:8),' ', & hh_init,':',mm_init,':',ss_init - status = nf90_put_att(ncid,varid,'units',title) - call ice_check_nc(status, subname// ' ERROR: time units', & - file=__FILE__, line=__LINE__) if (days_per_year == 360) then - status = nf90_put_att(ncid,varid,'calendar','360_day') - call ice_check_nc(status, subname// ' ERROR: time calendar 360', & - file=__FILE__, line=__LINE__) + cal_att='360_day' elseif (days_per_year == 365 .and. .not.use_leap_years ) then - status = nf90_put_att(ncid,varid,'calendar','noleap') - call ice_check_nc(status, subname// ' ERROR: time calendar noleap', & - file=__FILE__, line=__LINE__) + cal_att='noleap' elseif (use_leap_years) then - status = nf90_put_att(ncid,varid,'calendar','Gregorian') - call ice_check_nc(status, subname// ' ERROR: time calendar Gregorian', & - file=__FILE__, line=__LINE__) + cal_att='Gregorian' else call abort_ice(subname//' ERROR: invalid calendar settings', file=__FILE__, line=__LINE__) endif + time_coord = coord_attributes('time', 'time', trim(cal_units)) + call ice_hist_coord_def(ncid, time_coord, nf90_double, (/timid/), varid) + + status = nf90_put_att(ncid,varid,'calendar',cal_att) !extra attribute + call ice_check_nc(status, subname//' ERROR: defining att calendar: '//cal_att,file=__FILE__,line=__LINE__) if (hist_avg(ns) .and. .not. write_ic) then status = nf90_put_att(ncid,varid,'bounds','time_bounds') - call ice_check_nc(status, subname// ' ERROR: time bounds', & - file=__FILE__, line=__LINE__) + call ice_check_nc(status, subname//' ERROR: defining att bounds time_bounds',file=__FILE__,line=__LINE__) endif - !----------------------------------------------------------------- - ! Define attributes for time bounds if hist_avg is true - !----------------------------------------------------------------- - + ! Define coord time_bounds if hist_avg is true if (hist_avg(ns) .and. .not. write_ic) then + time_coord = coord_attributes('time_bounds', 'time interval endpoints', trim(cal_units)) + dimid(1) = boundid dimid(2) = timid - status = nf90_def_var(ncid,'time_bounds',lprecision,dimid(1:2),varid) - call ice_check_nc(status, subname// ' ERROR: defining var time_bounds', & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid,varid,'long_name', 'time interval endpoints') - call ice_check_nc(status, subname// ' ERROR: time_bounds long_name', & - file=__FILE__, line=__LINE__) - write(cdate,'(i8.8)') idate0 - write(title,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & - cdate(1:4),'-',cdate(5:6),'-',cdate(7:8),' ', & - hh_init,':',mm_init,':',ss_init - status = nf90_put_att(ncid,varid,'units',title) - call ice_check_nc(status, subname// ' ERROR: time_bounds units', & - file=__FILE__, line=__LINE__) - if (days_per_year == 360) then - status = nf90_put_att(ncid,varid,'calendar','360_day') - call ice_check_nc(status, subname// ' ERROR: time calendar 360 time bounds', & - file=__FILE__, line=__LINE__) - elseif (days_per_year == 365 .and. .not.use_leap_years ) then - status = nf90_put_att(ncid,varid,'calendar','noleap') - call ice_check_nc(status, subname// ' ERROR: time calendar noleap time bounds', & - file=__FILE__, line=__LINE__) - elseif (use_leap_years) then - status = nf90_put_att(ncid,varid,'calendar','Gregorian') - call ice_check_nc(status, subname// ' ERROR: time calendar Gregorian time bounds', & - file=__FILE__, line=__LINE__) - else - call abort_ice(subname//' ERROR: invalid calendar settings', file=__FILE__, line=__LINE__) - endif + call ice_hist_coord_def(ncid, time_coord, nf90_double, dimid(1:2), varid) + status = nf90_put_att(ncid,varid,'calendar',cal_att) + call ice_check_nc(status, subname//' ERROR: defining att calendar: '//cal_att,file=__FILE__,line=__LINE__) endif !----------------------------------------------------------------- @@ -431,16 +406,7 @@ subroutine ice_write_hist (ns) dimid(3) = timid do i = 1, ncoord - status = nf90_def_var(ncid, var_coord(i)%short_name, lprecision, & - dimid(1:2), varid) - call ice_check_nc(status, subname// ' ERROR: defining short_name for '//var_coord(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid,varid,'long_name',var_coord(i)%long_name) - call ice_check_nc(status, subname// ' ERROR: defining long_name for '//var_coord(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid, varid, 'units', var_coord(i)%units) - call ice_check_nc(status, subname// ' ERROR: defining units for '//var_coord(i)%short_name, & - file=__FILE__, line=__LINE__) + call ice_hist_coord_def(ncid, var_coord(i), lprecision, dimid(1:2), varid) call ice_write_hist_fill(ncid,varid,var_coord(i)%short_name,history_precision) if (var_coord(i)%short_name == 'ULAT') then status = nf90_put_att(ncid,varid,'comment', & @@ -465,31 +431,13 @@ subroutine ice_write_hist (ns) do i = 1, nvar_grdz if (igrdz(i)) then - status = nf90_def_var(ncid, var_grdz(i)%short_name, & - lprecision, dimidex(i), varid) - call ice_check_nc(status, subname// ' ERROR: defining short_name for '//var_grdz(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid,varid,'long_name',var_grdz(i)%long_name) - call ice_check_nc(status, subname// ' ERROR: defining long_name for '//var_grdz(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid, varid, 'units', var_grdz(i)%units) - call ice_check_nc(status, subname// ' ERROR: defining units for '//var_grdz(i)%short_name, & - file=__FILE__, line=__LINE__) + call ice_hist_coord_def(ncid, var_grdz(i), lprecision, dimidex(i:i), varid) endif enddo do i = 1, nvar_grd if (igrd(i)) then - status = nf90_def_var(ncid, var_grd(i)%req%short_name, & - lprecision, dimid(1:2), varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//var_grd(i)%req%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid,varid, 'long_name', var_grd(i)%req%long_name) - call ice_check_nc(status, subname// ' ERROR: defining long_name for '//var_grd(i)%req%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid, varid, 'units', var_grd(i)%req%units) - call ice_check_nc(status, subname// ' ERROR: defining units for '//var_grd(i)%req%short_name, & - file=__FILE__, line=__LINE__) + call ice_hist_coord_def(ncid, var_grd(i)%req, lprecision, dimid(1:2), varid) status = nf90_put_att(ncid, varid, 'coordinates', var_grd(i)%coordinates) call ice_check_nc(status, subname// ' ERROR: defining coordinates for '//var_grd(i)%req%short_name, & file=__FILE__, line=__LINE__) @@ -503,27 +451,18 @@ subroutine ice_write_hist (ns) dimid_nverts(3) = jmtid do i = 1, nvar_verts if (f_bounds) then - status = nf90_def_var(ncid, var_nverts(i)%short_name, & - lprecision,dimid_nverts, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//var_nverts(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid,varid, 'long_name', var_nverts(i)%long_name) - call ice_check_nc(status, subname// ' ERROR: defining long_name for '//var_nverts(i)%short_name, & - file=__FILE__, line=__LINE__) - status = nf90_put_att(ncid, varid, 'units', var_nverts(i)%units) - call ice_check_nc(status, subname// ' ERROR: defining units for '//var_nverts(i)%short_name, & - file=__FILE__, line=__LINE__) + call ice_hist_coord_def(ncid, var_nverts(i), lprecision, dimid_nverts, varid) call ice_write_hist_fill(ncid,varid,var_nverts(i)%short_name,history_precision) endif enddo + !----------------------------------------------------------------- + ! define attributes for time-variant variables + !----------------------------------------------------------------- + do n=1,num_avail_hist_fields_2D if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimid, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimid,ns) endif enddo ! num_avail_hist_fields_2D @@ -534,11 +473,7 @@ subroutine ice_write_hist (ns) do n = n2D + 1, n3Dccum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimidz, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Dc @@ -549,11 +484,7 @@ subroutine ice_write_hist (ns) do n = n3Dccum + 1, n3Dzcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimidz, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Dz @@ -564,11 +495,7 @@ subroutine ice_write_hist (ns) do n = n3Dzcum + 1, n3Dbcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimidz, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Db @@ -579,11 +506,7 @@ subroutine ice_write_hist (ns) do n = n3Dbcum + 1, n3Dacum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimidz, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Da @@ -594,11 +517,7 @@ subroutine ice_write_hist (ns) do n = n3Dacum + 1, n3Dfcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & - lprecision, dimidz, varid) - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Df @@ -610,12 +529,7 @@ subroutine ice_write_hist (ns) do n = n3Dfcum + 1, n4Dicum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & -! lprecision, dimidcz, varid) - lprecision, dimidcz(1:4), varid) ! ferret - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidcz,ns) endif enddo ! num_avail_hist_fields_4Di @@ -627,12 +541,7 @@ subroutine ice_write_hist (ns) do n = n4Dicum + 1, n4Dscum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & -! lprecision, dimidcz, varid) - lprecision, dimidcz(1:4), varid) ! ferret - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, dimidcz,ns) endif enddo ! num_avail_hist_fields_4Ds @@ -644,13 +553,10 @@ subroutine ice_write_hist (ns) do n = n4Dscum + 1, n4Dfcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - status = nf90_def_var(ncid, avail_hist_fields(n)%vname, & -! lprecision, dimidcz, varid) - lprecision, dimidcz(1:4), varid) ! ferret - call ice_check_nc(status, subname// ' ERROR: defining variable '//avail_hist_fields(n)%vname, & - file=__FILE__, line=__LINE__) - call ice_write_hist_attrs(ncid,varid,avail_hist_fields(n),ns) - endif + call ice_hist_field_def(ncid, avail_hist_fields(n),lprecision, & + ! dimidcz, ns) + dimidcz(1:4),ns) ! ferret + endif enddo ! num_avail_hist_fields_4Df !----------------------------------------------------------------- @@ -1260,28 +1166,46 @@ subroutine ice_write_hist (ns) end subroutine ice_write_hist !======================================================================= +! Defines a (time-dependent) history var in the history file +! variables have short_name, long_name and units, coordiantes and cell_measures attributes, +! and are compressed and chunked for 'hdf5' - subroutine ice_write_hist_attrs(ncid, varid, hfield, ns) + subroutine ice_hist_field_def(ncid, hfield, lprecision, dimids, ns) - use ice_kinds_mod + use ice_history_shared, only: history_deflate, history_chunksize, history_format, ice_hist_field, & + history_precision, hist_avg use ice_calendar, only: histfreq, histfreq_n, write_ic - use ice_history_shared, only: ice_hist_field, history_precision, & - hist_avg -#ifdef USE_NETCDF - use netcdf -#endif - integer (kind=int_kind), intent(in) :: ncid ! netcdf file id - integer (kind=int_kind), intent(in) :: varid ! netcdf variable id - type (ice_hist_field) , intent(in) :: hfield ! history file info - integer (kind=int_kind), intent(in) :: ns ! history stream + integer(kind=int_kind), intent(in) :: ncid, dimids(:), lprecision, ns + type(ice_hist_field), intent(in) :: hfield - ! local variables + !local vars + integer(kind=int_kind) :: chunks(size(dimids)), i, status, varid - integer (kind=int_kind) :: status - character(len=*), parameter :: subname = '(ice_write_hist_attrs)' + character(len=*), parameter :: subname = '(ice_hist_field_def)' #ifdef USE_NETCDF + status = nf90_def_var(ncid, hfield%vname, lprecision, dimids, varid) + call ice_check_nc(status, subname//' ERROR: defining var '//trim(hfield%vname),file=__FILE__,line=__LINE__) + + if (history_format=='hdf5' .and. size(dimids)>1) then + if (dimids(1)==imtid .and. dimids(2)==jmtid) then + chunks(1)=history_chunksize(1) + chunks(2)=history_chunksize(2) + do i = 3, size(dimids) + chunks(i) = 0 + enddo + status = nf90_def_var_chunking(ncid, varid, NF90_CHUNKED, chunksizes=chunks) + call ice_check_nc(status, subname//' ERROR chunking var '//trim(hfield%vname), file=__FILE__, line=__LINE__) + endif + endif + + if (history_format=='hdf5' .and. history_deflate/=0) then + status = nf90_def_var_deflate(ncid, varid, shuffle=0, deflate=1, deflate_level=history_deflate) + call ice_check_nc(status, subname//' ERROR deflating var '//trim(hfield%vname), file=__FILE__, line=__LINE__) + endif + + ! add attributes status = nf90_put_att(ncid,varid,'units', hfield%vunit) call ice_check_nc(status, subname// ' ERROR: defining units for '//hfield%vname, & file=__FILE__, line=__LINE__) @@ -1346,17 +1270,13 @@ subroutine ice_write_hist_attrs(ncid, varid, hfield, ns) call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', file=__FILE__, line=__LINE__) #endif - end subroutine ice_write_hist_attrs + end subroutine ice_hist_field_def !======================================================================= +! Defines missing_value and _FillValue attributes subroutine ice_write_hist_fill(ncid,varid,vname,precision) - use ice_kinds_mod -#ifdef USE_NETCDF - use netcdf -#endif - integer (kind=int_kind), intent(in) :: ncid ! netcdf file id integer (kind=int_kind), intent(in) :: varid ! netcdf var id character(len=*), intent(in) :: vname ! var name @@ -1389,6 +1309,59 @@ subroutine ice_write_hist_fill(ncid,varid,vname,precision) end subroutine ice_write_hist_fill +!======================================================================= +! Defines a coordinate var in the history file +! coordinates have short_name, long_name and units attributes, +! and are compressed for 'hdf5' when more than one dimensional + + subroutine ice_hist_coord_def(ncid, coord, lprecision, dimids, varid) + + use ice_history_shared, only: history_deflate, history_format, history_chunksize + + integer(kind=int_kind), intent(in) :: ncid, dimids(:), lprecision + type(coord_attributes), intent(in) :: coord + integer(kind=int_kind), intent(inout) :: varid + + !local vars + integer(kind=int_kind) ::chunks(size(dimids)), i, status + + character(len=*), parameter :: subname = '(ice_hist_coord_def)' + +#ifdef USE_NETCDF + status = nf90_def_var(ncid, coord%short_name, lprecision, dimids, varid) + call ice_check_nc(status, subname//' ERROR: defining coord '//coord%short_name,file=__FILE__,line=__LINE__) + + if (history_format=='hdf5' .and. size(dimids)>1) then + if (dimids(1)==imtid .and. dimids(2)==jmtid) then + chunks(1)=history_chunksize(1) + chunks(2)=history_chunksize(2) + do i = 3, size(dimids) + chunks(i) = 0 + enddo + status = nf90_def_var_chunking(ncid, varid, NF90_CHUNKED, chunksizes=chunks) + call ice_check_nc(status, subname//' ERROR chunking var '//trim(coord%short_name), file=__FILE__, line=__LINE__) + endif + endif + + if (history_format=='hdf5' .and. history_deflate/=0) then + status=nf90_def_var_deflate(ncid, varid, shuffle=0, deflate=1, deflate_level=history_deflate) + call ice_check_nc(status, subname//' ERROR deflating var '//trim(coord%short_name), file=__FILE__, line=__LINE__) + endif + + status = nf90_put_att(ncid,varid,'long_name',trim(coord%long_name)) + call ice_check_nc(status, subname// ' ERROR: defining long_name for '//coord%short_name, & + file=__FILE__, line=__LINE__) + status = nf90_put_att(ncid, varid, 'units', trim(coord%units)) + call ice_check_nc(status, subname// ' ERROR: defining units for '//coord%short_name, & + file=__FILE__, line=__LINE__) + +#else + call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', & + file=__FILE__, line=__LINE__) +#endif + + end subroutine ice_hist_coord_def + !======================================================================= end module ice_history_write diff --git a/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_restart.F90 b/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_restart.F90 index c670bf016..e9be45481 100644 --- a/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_restart.F90 +++ b/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_restart.F90 @@ -18,7 +18,8 @@ module ice_restart use ice_read_write, only: ice_check_nc use ice_restart_shared, only: & restart_ext, restart_dir, restart_file, pointer_file, & - runid, use_restart_time, lcdf64, lenstr, restart_coszen + runid, use_restart_time, lenstr, restart_coszen, restart_format, & + restart_chunksize, restart_deflate use ice_fileunits, only: nu_diag, nu_rst_pointer use ice_exit, only: abort_ice use icepack_intfc, only: icepack_query_parameters @@ -29,10 +30,12 @@ module ice_restart implicit none private public :: init_restart_write, init_restart_read, & - read_restart_field, write_restart_field, final_restart, & - query_field + read_restart_field, write_restart_field, final_restart, & + query_field - integer (kind=int_kind) :: ncid + integer (kind=int_kind) :: ncid , & + dimid_ni, & ! netCDF identifiers + dimid_nj !======================================================================= @@ -169,8 +172,7 @@ subroutine init_restart_write(filename_spec) integer (kind=int_kind), allocatable :: dims(:) integer (kind=int_kind) :: & - dimid_ni, & ! netCDF identifiers - dimid_nj, & ! + dimid_ncat, & ! iflag, & ! netCDF creation flag status ! status variable from netCDF routine @@ -216,8 +218,18 @@ subroutine init_restart_write(filename_spec) write(nu_rst_pointer,'(a)') filename close(nu_rst_pointer) - iflag = 0 - if (lcdf64) iflag = nf90_64bit_offset + if (restart_format == 'cdf1') then + iflag = nf90_clobber + elseif (restart_format == 'cdf2') then + iflag = ior(nf90_clobber,nf90_64bit_offset) + elseif (restart_format == 'cdf5') then + iflag = ior(nf90_clobber,nf90_64bit_data) + elseif (restart_format == 'hdf5') then + iflag = ior(nf90_clobber,nf90_netcdf4) + else + call abort_ice(subname//' ERROR: restart_format not allowed for '//trim(restart_format), & + file=__FILE__, line=__LINE__) + endif status = nf90_create(trim(filename), iflag, ncid) call ice_check_nc(status, subname//' ERROR: creating '//trim(filename), file=__FILE__, line=__LINE__) @@ -873,14 +885,32 @@ subroutine define_rest_field(ncid, vname, dims) integer (kind=int_kind) :: varid - integer (kind=int_kind) :: & - status ! status variable from netCDF routine + integer (kind=int_kind) :: chunks(size(dims)), status, i character(len=*), parameter :: subname = '(define_rest_field)' #ifdef USE_NETCDF + status = nf90_def_var(ncid,trim(vname),nf90_double,dims,varid) call ice_check_nc(status, subname//' ERROR: def var '//trim(vname), file=__FILE__, line=__LINE__) + + if (restart_format=='hdf5' .and. size(dims)>1) then + if (dims(1)==dimid_ni .and. dims(2)==dimid_nj) then + chunks(1)=restart_chunksize(1) + chunks(2)=restart_chunksize(2) + do i = 3, size(dims) + chunks(i) = 0 + enddo + status = nf90_def_var_chunking(ncid, varid, NF90_CHUNKED, chunksizes=chunks) + call ice_check_nc(status, subname//' ERROR: chunking var '//trim(vname), file=__FILE__, line=__LINE__) + endif + endif + + if (restart_format=='hdf5' .and. restart_deflate/=0) then + status=nf90_def_var_deflate(ncid, varid, shuffle=0, deflate=1, deflate_level=restart_deflate) + call ice_check_nc(status, subname//' ERROR deflating var '//trim(vname), file=__FILE__, line=__LINE__) + endif + #else call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', & file=__FILE__, line=__LINE__) @@ -911,7 +941,7 @@ logical function query_field(nu,vname) endif call broadcast_scalar(query_field,master_task) #else - call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined for '//trim(ice_ic), & + call abort_ice(subname//' ERROR: USE_NETCDF cpp not defined', & file=__FILE__, line=__LINE__) #endif diff --git a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_history_write.F90 b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_history_write.F90 index bb4ef0ea1..daebe1f2e 100644 --- a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_history_write.F90 +++ b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_history_write.F90 @@ -23,11 +23,27 @@ module ice_history_write use ice_exit, only: abort_ice use icepack_intfc, only: icepack_warnings_flush, icepack_warnings_aborted use icepack_intfc, only: icepack_query_parameters + use ice_calendar, only: write_ic, histfreq + use ice_pio implicit none private + + TYPE coord_attributes ! netcdf coordinate attributes + character (len=11) :: short_name + character (len=45) :: long_name + character (len=30) :: units + END TYPE coord_attributes + + TYPE req_attributes ! req'd netcdf attributes + type (coord_attributes) :: req + character (len=20) :: coordinates + END TYPE req_attributes + public :: ice_write_hist + integer (kind=int_kind) :: imtid,jmtid + !======================================================================= contains @@ -42,8 +58,8 @@ subroutine ice_write_hist (ns) use ice_blocks, only: nx_block, ny_block use ice_broadcast, only: broadcast_scalar - use ice_calendar, only: msec, timesecs, idate, idate0, write_ic, & - histfreq, histfreq_n, days_per_year, use_leap_years, dayyr, & + use ice_calendar, only: msec, timesecs, idate, idate0, & + histfreq_n, days_per_year, use_leap_years, dayyr, & hh_init, mm_init, ss_init use ice_communicate, only: my_task, master_task use ice_domain, only: distrb_info, nblocks @@ -57,8 +73,7 @@ subroutine ice_write_hist (ns) lonn_bounds, latn_bounds, lone_bounds, late_bounds use ice_history_shared use ice_arrays_column, only: hin_max, floe_rad_c - use ice_restart_shared, only: runid, lcdf64 - use ice_pio + use ice_restart_shared, only: runid use pio integer (kind=int_kind), intent(in) :: ns @@ -66,7 +81,7 @@ subroutine ice_write_hist (ns) ! local variables integer (kind=int_kind) :: i,j,k,ic,n,nn, & - ncid,status,imtid,jmtid,kmtidi,kmtids,kmtidb, cmtid,timid, & + ncid,status,kmtidi,kmtids,kmtidb, cmtid,timid, & length,nvertexid,ivertex,kmtida,fmtid integer (kind=int_kind), dimension(2) :: dimid2 integer (kind=int_kind), dimension(3) :: dimid3 @@ -75,16 +90,15 @@ subroutine ice_write_hist (ns) integer (kind=int_kind), dimension(3) :: dimid_nverts integer (kind=int_kind), dimension(6) :: dimidex real (kind= dbl_kind) :: ltime2 - character (char_len) :: title - character (char_len) :: time_period_freq = 'none' - character (char_len_long) :: ncfile(max_nstrm) - integer (kind=int_kind) :: iotype + character (len=8) :: cdate + character (len=char_len_long) :: title, cal_units, cal_att + character (len=char_len) :: time_period_freq = 'none' + character (len=char_len_long) :: ncfile(max_nstrm) - integer (kind=int_kind) :: icategory,ind,i_aice,boundid + integer (kind=int_kind) :: icategory,ind,i_aice,boundid, lprecision - character (char_len) :: start_time,current_date,current_time + character (len=char_len) :: start_time,current_date,current_time character (len=16) :: c_aice - character (len=8) :: cdate type(file_desc_t) :: File type(io_desc_t) :: iodesc2d, & @@ -93,6 +107,9 @@ subroutine ice_write_hist (ns) iodesc4di, iodesc4ds, iodesc4df type(var_desc_t) :: varid + ! time coord + TYPE(coord_attributes) :: time_coord + ! 8 coordinate variables: TLON, TLAT, ULON, ULAT, NLON, NLAT, ELON, ELAT INTEGER (kind=int_kind), PARAMETER :: ncoord = 8 @@ -104,17 +121,6 @@ subroutine ice_write_hist (ns) ! lonn_bounds, latn_bounds, lone_bounds, late_bounds INTEGER (kind=int_kind), PARAMETER :: nvar_verts = 8 - TYPE coord_attributes ! netcdf coordinate attributes - character (len=11) :: short_name - character (len=45) :: long_name - character (len=20) :: units - END TYPE coord_attributes - - TYPE req_attributes ! req'd netcdf attributes - type (coord_attributes) :: req - character (len=20) :: coordinates - END TYPE req_attributes - TYPE(req_attributes), dimension(nvar_grd) :: var_grd TYPE(coord_attributes), dimension(ncoord) :: var_coord TYPE(coord_attributes), dimension(nvar_verts) :: var_nverts @@ -131,8 +137,7 @@ subroutine ice_write_hist (ns) real (kind=real_kind), allocatable :: workr4(:,:,:,:,:) real (kind=real_kind), allocatable :: workr3v(:,:,:,:) - character(len=char_len_long) :: & - filename + character(len=char_len_long) :: filename integer (kind=int_kind), dimension(1) :: & tim_start,tim_length ! dimension quantities for netCDF @@ -143,7 +148,7 @@ subroutine ice_write_hist (ns) real (kind=dbl_kind) :: secday real (kind=dbl_kind) :: rad_to_deg - integer (kind=int_kind) :: lprecision + logical (kind=log_kind), save :: first_call = .true. character(len=*), parameter :: subname = '(ice_write_hist)' @@ -167,11 +172,10 @@ subroutine ice_write_hist (ns) call broadcast_scalar(filename, master_task) ! create file - iotype = PIO_IOTYPE_NETCDF - if (history_format == 'pio_pnetcdf') iotype = PIO_IOTYPE_PNETCDF File%fh=-1 call ice_pio_init(mode='write', filename=trim(filename), File=File, & - clobber=.true., cdf64=lcdf64, iotype=iotype) + clobber=.true., fformat=trim(history_format), rearr=trim(history_rearranger), & + iotasks=history_iotasks, root=history_root, stride=history_stride, debug=first_call) call ice_pio_initdecomp(iodesc=iodesc2d, precision=history_precision) call ice_pio_initdecomp(ndim3=ncat_hist, iodesc=iodesc3dc, precision=history_precision) @@ -232,64 +236,40 @@ subroutine ice_write_hist (ns) ! define coordinate variables: time, time_bounds !----------------------------------------------------------------- - call ice_pio_check(pio_def_var(File,'time',pio_double,(/timid/),varid), & - subname//' ERROR: defining var time',file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File,varid,'long_name','time'), & - subname//' ERROR: defining att long_name time',file=__FILE__,line=__LINE__) - write(cdate,'(i8.8)') idate0 - write(title,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & + write(cal_units,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & cdate(1:4),'-',cdate(5:6),'-',cdate(7:8),' ', & hh_init,':',mm_init,':',ss_init - call ice_pio_check(pio_put_att(File,varid,'units',trim(title)), & - subname//' ERROR: defining att units '//trim(title),file=__FILE__,line=__LINE__) if (days_per_year == 360) then - call ice_pio_check(pio_put_att(File,varid,'calendar','360_day'), & - subname//' ERROR: defining att calendar 360',file=__FILE__,line=__LINE__) + cal_att='360_day' elseif (days_per_year == 365 .and. .not.use_leap_years ) then - call ice_pio_check(pio_put_att(File,varid,'calendar','noleap'), & - subname//' ERROR: defining att calendar noleap',file=__FILE__,line=__LINE__) + cal_att='noleap' elseif (use_leap_years) then - call ice_pio_check(pio_put_att(File,varid,'calendar','Gregorian'), & - subname//' ERROR: defining att calendar Gregorian',file=__FILE__,line=__LINE__) + cal_att='Gregorian' else call abort_ice(subname//' ERROR: invalid calendar settings') endif + time_coord = coord_attributes('time', 'time', trim(cal_units)) + call ice_hist_coord_def(File, time_coord, pio_double, (/timid/), varid) + call ice_pio_check(pio_put_att(File,varid,'calendar',cal_att), & + subname//' ERROR: defining att calendar: '//cal_att,file=__FILE__,line=__LINE__) if (hist_avg(ns) .and. .not. write_ic) then call ice_pio_check(pio_put_att(File,varid,'bounds','time_bounds'), & subname//' ERROR: defining att bounds time_bounds',file=__FILE__,line=__LINE__) endif - ! Define attributes for time_bounds if hist_avg is true + ! Define coord time_bounds if hist_avg is true if (hist_avg(ns) .and. .not. write_ic) then + time_coord = coord_attributes('time_bounds', 'time interval endpoints', trim(cal_units)) + dimid2(1) = boundid dimid2(2) = timid - call ice_pio_check(pio_def_var(File,'time_bounds',pio_double,dimid2,varid), & - subname//' ERROR: defining var time_bounds',file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File,varid,'long_name', 'time interval endpoints'), & - subname//' ERROR: defining att long_name time interval endpoints',file=__FILE__,line=__LINE__) - - if (days_per_year == 360) then - call ice_pio_check(pio_put_att(File,varid,'calendar','360_day'), & - subname//' ERROR: defining att calendar 360 time bounds',file=__FILE__,line=__LINE__) - elseif (days_per_year == 365 .and. .not.use_leap_years ) then - call ice_pio_check(pio_put_att(File,varid,'calendar','noleap'), & - subname//' ERROR: defining att calendar noleap time bounds',file=__FILE__,line=__LINE__) - elseif (use_leap_years) then - call ice_pio_check(pio_put_att(File,varid,'calendar','Gregorian'), & - subname//' ERROR: defining att calendar Gregorian time bounds',file=__FILE__,line=__LINE__) - else - call abort_ice(subname//' ERROR: invalid calendar settings') - endif - write(cdate,'(i8.8)') idate0 - write(title,'(a,a4,a1,a2,a1,a2,a1,i2.2,a1,i2.2,a1,i2.2)') 'days since ', & - cdate(1:4),'-',cdate(5:6),'-',cdate(7:8),' ', & - hh_init,':',mm_init,':',ss_init - call ice_pio_check(pio_put_att(File,varid,'units',trim(title)), & - subname//' ERROR: defining att units '//trim(title),file=__FILE__,line=__LINE__) + call ice_hist_coord_def(File, time_coord, pio_double, dimid2, varid) + call ice_pio_check(pio_put_att(File,varid,'calendar',cal_att), & + subname//' ERROR: defining att calendar: '//cal_att,file=__FILE__,line=__LINE__) endif !----------------------------------------------------------------- @@ -438,12 +418,7 @@ subroutine ice_write_hist (ns) dimid2(2) = jmtid do i = 1, ncoord - call ice_pio_check(pio_def_var(File, trim(var_coord(i)%short_name), lprecision,dimid2, varid), & - subname//' ERROR: defining var '//trim(var_coord(i)%short_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File,varid,'long_name',trim(var_coord(i)%long_name)), & - subname//' ERROR: defining att long_name '//trim(var_coord(i)%long_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File, varid, 'units', trim(var_coord(i)%units)), & - subname//' ERROR: defining att units '//trim(var_coord(i)%units),file=__FILE__,line=__LINE__) + call ice_hist_coord_def(File, var_coord(i), lprecision, dimid2, varid) call ice_write_hist_fill(File,varid,var_coord(i)%short_name,history_precision) if (var_coord(i)%short_name == 'ULAT') then call ice_pio_check(pio_put_att(File,varid,'comment', & @@ -466,23 +441,13 @@ subroutine ice_write_hist (ns) do i = 1, nvar_grdz if (igrdz(i)) then - call ice_pio_check(pio_def_var(File, trim(var_grdz(i)%short_name), lprecision,(/dimidex(i)/), varid), & - subname//' ERROR: defining var'//trim(var_grdz(i)%short_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File, varid, 'long_name', var_grdz(i)%long_name), & - subname//' ERROR: defining att long_name '//trim(var_grdz(i)%long_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File, varid, 'units' , var_grdz(i)%units), & - subname//' ERROR: defining att units '//trim(var_grdz(i)%units),file=__FILE__,line=__LINE__) + call ice_hist_coord_def(File, var_grdz(i), lprecision, dimidex(i:i), varid) endif enddo do i = 1, nvar_grd if (igrd(i)) then - call ice_pio_check(pio_def_var(File, trim(var_grd(i)%req%short_name), lprecision, dimid2, varid), & - subname//' ERROR: defining var'//trim(var_grd(i)%req%short_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File,varid, 'long_name', trim(var_grd(i)%req%long_name)), & - subname//' ERROR: defining att long_name '//trim(var_grd(i)%req%long_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File, varid, 'units', trim(var_grd(i)%req%units)), & - subname//' ERROR: defining att units '//trim(var_grd(i)%req%units),file=__FILE__,line=__LINE__) + call ice_hist_coord_def(File, var_grd(i)%req, lprecision, dimid2, varid) call ice_pio_check(pio_put_att(File, varid, 'coordinates', trim(var_grd(i)%coordinates)), & subname//' ERROR: defining att coordinates '//trim(var_grd(i)%coordinates),file=__FILE__,line=__LINE__) call ice_write_hist_fill(File,varid,var_grd(i)%req%short_name,history_precision) @@ -495,12 +460,7 @@ subroutine ice_write_hist (ns) dimid_nverts(3) = jmtid do i = 1, nvar_verts if (f_bounds) then - call ice_pio_check(pio_def_var(File, trim(var_nverts(i)%short_name),lprecision,dimid_nverts, varid), & - subname//' ERROR: defining var'//trim(var_nverts(i)%short_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File,varid, 'long_name', trim(var_nverts(i)%long_name)), & - subname//' ERROR: defining att long_name '//trim(var_nverts(i)%long_name),file=__FILE__,line=__LINE__) - call ice_pio_check(pio_put_att(File, varid, 'units', trim(var_nverts(i)%units)), & - subname//' ERROR: defining att units '//trim(var_nverts(i)%units),file=__FILE__,line=__LINE__) + call ice_hist_coord_def(File, var_nverts(i), lprecision, dimid_nverts, varid) call ice_write_hist_fill(File,varid,var_nverts(i)%short_name,history_precision) endif enddo @@ -509,26 +469,18 @@ subroutine ice_write_hist (ns) ! define attributes for time-variant variables !----------------------------------------------------------------- - !----------------------------------------------------------------- ! 2D - !----------------------------------------------------------------- - dimid3(1) = imtid dimid3(2) = jmtid dimid3(3) = timid do n=1,num_avail_hist_fields_2D if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimid3, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimid3, ns) endif - enddo ! num_avail_hist_fields_2D + enddo - !----------------------------------------------------------------- ! 3D (category) - !----------------------------------------------------------------- - dimidz(1) = imtid dimidz(2) = jmtid dimidz(3) = cmtid @@ -536,16 +488,11 @@ subroutine ice_write_hist (ns) do n = n2D + 1, n3Dccum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Dc - !----------------------------------------------------------------- ! 3D (ice layers) - !----------------------------------------------------------------- - dimidz(1) = imtid dimidz(2) = jmtid dimidz(3) = kmtidi @@ -553,16 +500,11 @@ subroutine ice_write_hist (ns) do n = n3Dccum + 1, n3Dzcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Dz - !----------------------------------------------------------------- ! 3D (biology ice layers) - !----------------------------------------------------------------- - dimidz(1) = imtid dimidz(2) = jmtid dimidz(3) = kmtidb @@ -570,16 +512,11 @@ subroutine ice_write_hist (ns) do n = n3Dzcum + 1, n3Dbcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Db - !----------------------------------------------------------------- ! 3D (biology snow layers) - !----------------------------------------------------------------- - dimidz(1) = imtid dimidz(2) = jmtid dimidz(3) = kmtida @@ -587,16 +524,11 @@ subroutine ice_write_hist (ns) do n = n3Dbcum + 1, n3Dacum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Da - !----------------------------------------------------------------- ! 3D (fsd) - !----------------------------------------------------------------- - dimidz(1) = imtid dimidz(2) = jmtid dimidz(3) = fmtid @@ -604,21 +536,11 @@ subroutine ice_write_hist (ns) do n = n3Dacum + 1, n3Dfcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidz,ns) endif enddo ! num_avail_hist_fields_3Df - !----------------------------------------------------------------- - ! define attributes for 4D variables - ! time coordinate is dropped - !----------------------------------------------------------------- - - !----------------------------------------------------------------- ! 4D (ice categories) - !----------------------------------------------------------------- - dimidcz(1) = imtid dimidcz(2) = jmtid dimidcz(3) = kmtidi @@ -627,16 +549,11 @@ subroutine ice_write_hist (ns) do n = n3Dfcum + 1, n4Dicum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidcz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidcz,ns) endif enddo ! num_avail_hist_fields_4Di - !----------------------------------------------------------------- ! 4D (snow layers) - !----------------------------------------------------------------- - dimidcz(1) = imtid dimidcz(2) = jmtid dimidcz(3) = kmtids @@ -645,16 +562,11 @@ subroutine ice_write_hist (ns) do n = n4Dicum + 1, n4Dscum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidcz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidcz,ns) endif enddo ! num_avail_hist_fields_4Ds - !----------------------------------------------------------------- ! 4D (fsd layers) - !----------------------------------------------------------------- - dimidcz(1) = imtid dimidcz(2) = jmtid dimidcz(3) = fmtid @@ -663,9 +575,7 @@ subroutine ice_write_hist (ns) do n = n4Dscum + 1, n4Dfcum if (avail_hist_fields(n)%vhistfreq == histfreq(ns) .or. write_ic) then - call ice_pio_check(pio_def_var(File, trim(avail_hist_fields(n)%vname), lprecision, dimidcz, varid), & - subname//' ERROR: defining var'//trim(avail_hist_fields(n)%vname),file=__FILE__,line=__LINE__) - call ice_write_hist_attrs(File,varid,avail_hist_fields(n),ns) + call ice_hist_field_def(File, avail_hist_fields(n),lprecision, dimidcz,ns) endif enddo ! num_avail_hist_fields_4Df @@ -686,7 +596,7 @@ subroutine ice_write_hist (ns) call ice_pio_check(pio_put_att(File,pio_global,'contents',trim(title)), & subname//' ERROR: defining att contents '//trim(title),file=__FILE__,line=__LINE__) - write(title,'(2a)') 'Los Alamos Sea Ice Model, ', trim(version_name) + write(title,'(2a)') 'CICE Sea Ice Model, ', trim(version_name) call ice_pio_check(pio_put_att(File,pio_global,'source',trim(title)), & subname//' ERROR: defining att source '//trim(title),file=__FILE__,line=__LINE__) @@ -741,13 +651,13 @@ subroutine ice_write_hist (ns) call ice_pio_check(pio_put_att(File,pio_global,'history',trim(start_time)), & subname//' ERROR: defining att history '//trim(start_time),file=__FILE__,line=__LINE__) - if (history_format == 'pio_pnetcdf') then - call ice_pio_check(pio_put_att(File,pio_global,'io_flavor','io_pio pnetcdf'), & - subname//' ERROR: defining att io_flavor',file=__FILE__,line=__LINE__) - else - call ice_pio_check(pio_put_att(File,pio_global,'io_flavor','io_pio netcdf'), & - subname//' ERROR: defining att io_flavor',file=__FILE__,line=__LINE__) - endif +#ifdef USE_PIO1 + call ice_pio_check(pio_put_att(File,pio_global,'io_flavor','io_pio1 '//trim(history_format)), & + subname//' ERROR: defining att io_flavor',file=__FILE__,line=__LINE__) +#else + call ice_pio_check(pio_put_att(File,pio_global,'io_flavor','io_pio2 '//trim(history_format)), & + subname//' ERROR: defining att io_flavor',file=__FILE__,line=__LINE__) +#endif !----------------------------------------------------------------- ! end define mode @@ -1315,7 +1225,6 @@ subroutine ice_write_hist (ns) ! similarly for num_avail_hist_fields_4Db (define workd4b, iodesc4db) - !----------------------------------------------------------------- ! clean-up PIO descriptors !----------------------------------------------------------------- @@ -1342,27 +1251,119 @@ subroutine ice_write_hist (ns) write(nu_diag,*) 'Finished writing ',trim(ncfile(ns)) endif + first_call = .false. + end subroutine ice_write_hist + !======================================================================= +! Defines a coordinate var in the history file +! coordinates have short_name, long_name and units attributes, +! and are compressed for 'hdf5' when more than one dimensional + + subroutine ice_hist_coord_def(File, coord,lprecision, dimids,varid) + + use pio, only: file_desc_t, var_desc_t, pio_def_var, pio_put_att +#ifndef USE_PIO1 + use pio, only: pio_def_var_deflate + use pio_nf, only: pio_def_var_chunking !This is missing from pio module <2.6.0 + use netcdf, only: NF90_CHUNKED + use ice_history_shared, only: history_deflate, history_chunksize, history_format +#endif + + type(file_desc_t), intent(inout) :: File + type(coord_attributes), intent(in) :: coord + integer(kind=int_kind), intent(in) :: dimids(:), lprecision + type(var_desc_t), intent(inout) :: varid + + ! local vars + integer(kind=int_kind) :: chunks(size(dimids)), i, status + + character(len=*), parameter :: subname = '(ice_hist_coord_def)' + + !define var, set deflate, long_name and units + status = pio_def_var(File, coord%short_name, lprecision, dimids, varid) + call ice_pio_check(status, & + subname//' ERROR: defining coord '//coord%short_name,file=__FILE__,line=__LINE__) +#ifndef USE_PIO1 + if (history_deflate/=0 .and. history_format=='hdf5') then + status = pio_def_var_deflate(File, varid, shuffle=0, deflate=1, deflate_level=history_deflate) + call ice_pio_check(status, & + subname//' ERROR: deflating coord '//coord%short_name,file=__FILE__,line=__LINE__) + endif - subroutine ice_write_hist_attrs(File, varid, hfield, ns) + if (history_format=='hdf5' .and. size(dimids)>1) then + if (dimids(1)==imtid .and. dimids(2)==jmtid) then + chunks(1)=history_chunksize(1) + chunks(2)=history_chunksize(2) + do i = 3, size(dimids) + chunks(i) = 0 + enddo + status = pio_def_var_chunking(File, varid, NF90_CHUNKED, chunks) + call ice_pio_check(status, & + subname//' ERROR: chunking coord '//coord%short_name,file=__FILE__,line=__LINE__) + endif + endif +#endif + call ice_pio_check(pio_put_att(File,varid,'long_name',trim(coord%long_name)), & + subname//' ERROR: defining att long_name '//coord%long_name,file=__FILE__,line=__LINE__) + call ice_pio_check(pio_put_att(File, varid, 'units', trim(coord%units)), & + subname//' ERROR: defining att units '//coord%units,file=__FILE__,line=__LINE__) + + end subroutine ice_hist_coord_def +!======================================================================= +! Defines a (time-dependent) history var in the history file +! variables have short_name, long_name and units, coordiantes and cell_measures attributes, +! and are compressed and chunked for 'hdf5' + + subroutine ice_hist_field_def(File, hfield,lprecision, dimids, ns) + + use pio, only: file_desc_t , var_desc_t, pio_def_var, pio_put_att +#ifndef USE_PIO1 + use pio, only: pio_def_var_deflate + use pio_nf, only: pio_def_var_chunking !This is missing from pio module <2.6.0 + use netcdf, only: NF90_CHUNKED + use ice_history_shared, only: history_deflate, history_chunksize, history_format +#endif + use ice_history_shared, only: ice_hist_field, history_precision, hist_avg use ice_calendar, only: histfreq, histfreq_n, write_ic - use ice_history_shared, only: ice_hist_field, history_precision, & - hist_avg - use ice_pio - use pio - type(file_desc_t) :: File ! file id - type(var_desc_t) :: varid ! variable id - type (ice_hist_field), intent(in) :: hfield ! history file info - integer (kind=int_kind), intent(in) :: ns + type(file_desc_t), intent(inout) :: File + type(ice_hist_field) , intent(in) :: hfield + integer(kind=int_kind), intent(in) :: dimids(:), lprecision, ns - ! local variables + ! local vars + type(var_desc_t) :: varid + integer(kind=int_kind) :: chunks(size(dimids)), i, status - integer (kind=int_kind) :: status - character(len=*), parameter :: subname = '(ice_write_hist_attrs)' + character(len=*), parameter :: subname = '(ice_hist_field_def)' + + status = pio_def_var(File, hfield%vname, lprecision, dimids, varid) + call ice_pio_check(status, & + subname//' ERROR: defining var '//hfield%vname,file=__FILE__,line=__LINE__) + +#ifndef USE_PIO1 + if (history_deflate/=0 .and. history_format=='hdf5') then + status = pio_def_var_deflate(File, varid, shuffle=0, deflate=1, deflate_level=history_deflate) + call ice_pio_check(status, & + subname//' ERROR: deflating var '//hfield%vname,file=__FILE__,line=__LINE__) + endif + + if (history_format=='hdf5' .and. size(dimids)>1) then + if (dimids(1)==imtid .and. dimids(2)==jmtid) then + chunks(1)=history_chunksize(1) + chunks(2)=history_chunksize(2) + do i = 3, size(dimids) + chunks(i) = 0 + enddo + status = pio_def_var_chunking(File, varid, NF90_CHUNKED, chunks) + call ice_pio_check(status, subname//' ERROR: chunking var '//hfield%vname,file=__FILE__,line=__LINE__) + endif + endif +#endif + + !var attributes call ice_pio_check(pio_put_att(File,varid,'units', trim(hfield%vunit)), & subname//' ERROR: defining att units '//trim(hfield%vunit),file=__FILE__,line=__LINE__) @@ -1418,17 +1419,17 @@ subroutine ice_write_hist_attrs(File, varid, hfield, ns) subname//' ERROR: defining att time_rep a',file=__FILE__,line=__LINE__) endif - end subroutine ice_write_hist_attrs + end subroutine ice_hist_field_def !======================================================================= +! Defines missing_value and _FillValue attributes subroutine ice_write_hist_fill(File,varid,vname,precision) - use ice_pio, only: ice_pio_check use pio, only: pio_put_att, file_desc_t, var_desc_t - type(file_desc_t) , intent(inout) :: File - type(var_desc_t) , intent(in) :: varid + type(file_desc_t), intent(inout) :: File + type(var_desc_t), intent(in) :: varid character(len=*), intent(in) :: vname integer (kind=int_kind), intent(in) :: precision diff --git a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90 b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90 index 8b02fb75e..565e7adbb 100644 --- a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90 +++ b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90 @@ -44,10 +44,11 @@ module ice_pio ! Initialize the io subsystem ! 2009-Feb-17 - J. Edwards - initial version - subroutine ice_pio_init(mode, filename, File, clobber, cdf64, iotype) + subroutine ice_pio_init(mode, filename, File, clobber, fformat, & + rearr, iotasks, root, stride, debug) #ifdef CESMCOUPLED - use shr_pio_mod, only: shr_pio_getiosys, shr_pio_getiotype + use shr_pio_mod, only: shr_pio_getiosys, shr_pio_getiotype, shr_pio_getioformat #else #ifdef GPTL use perf_mod, only : t_initf @@ -59,22 +60,31 @@ subroutine ice_pio_init(mode, filename, File, clobber, cdf64, iotype) character(len=*) , intent(in), optional :: filename type(file_desc_t) , intent(inout), optional :: File logical , intent(in), optional :: clobber - logical , intent(in), optional :: cdf64 - integer , intent(in), optional :: iotype + character(len=*) , intent(in), optional :: fformat + character(len=*) , intent(in), optional :: rearr + integer , intent(in), optional :: iotasks + integer , intent(in), optional :: root + integer , intent(in), optional :: stride + logical , intent(in), optional :: debug ! local variables integer (int_kind) :: & nml_error ! namelist read error flag - integer :: nprocs , istride, basetask, numiotasks, rearranger, pio_iotype, status, nmode - logical :: lclobber, lcdf64, exists - logical, save :: first_call = .true. + integer :: nprocs , lstride, lroot, liotasks, rearranger + integer :: pio_iotype, status, nmode0, nmode + logical :: lclobber, exists, ldebug character(len=*), parameter :: subname = '(ice_pio_init)' #ifdef CESMCOUPLED ice_pio_subsystem => shr_pio_getiosys(inst_name) pio_iotype = shr_pio_getiotype(inst_name) + if ((pio_iotype==PIO_IOTYPE_NETCDF).or.(pio_iotype==PIO_IOTYPE_PNETCDF)) then + nmode0 = shr_pio_getioformat(inst_name) + else + nmode=0 + endif call pio_seterrorhandling(ice_pio_subsystem, PIO_RETURN_ERROR) #else @@ -86,74 +96,113 @@ subroutine ice_pio_init(mode, filename, File, clobber, cdf64, iotype) #endif !--- initialize type of io - !pio_iotype = PIO_IOTYPE_PNETCDF - !pio_iotype = PIO_IOTYPE_NETCDF4C - !pio_iotype = PIO_IOTYPE_NETCDF4P - pio_iotype = PIO_IOTYPE_NETCDF - if (present(iotype)) then - pio_iotype = iotype + ldebug = .false. + if (present(debug)) then + ldebug = debug + endif + + if (present(fformat)) then + if (fformat(1:3) == 'cdf') then + pio_iotype = PIO_IOTYPE_NETCDF + elseif (fformat(1:3) == 'hdf') then + pio_iotype = PIO_IOTYPE_NETCDF4P + elseif (fformat(1:7) == 'pnetcdf') then + pio_iotype = PIO_IOTYPE_PNETCDF + else + call abort_ice(subname//' ERROR: format not allowed for '//trim(fformat), & + file=__FILE__, line=__LINE__) + endif + + if (fformat == 'cdf2' .or. fformat == 'pnetcdf2') then + nmode0 = PIO_64BIT_OFFSET + elseif (fformat == 'cdf5' .or. fformat == 'pnetcdf5') then + nmode0 = PIO_64BIT_DATA + else + nmode0 = 0 + endif + else + pio_iotype = PIO_IOTYPE_NETCDF + nmode0 = 0 + endif + + if (present(rearr)) then + if (rearr == 'box' .or. rearr == 'default') then + rearranger = PIO_REARR_BOX + elseif (rearr == 'subset') then + rearranger = PIO_REARR_SUBSET + else + call abort_ice(subname//' ERROR: rearr not allowed for '//trim(rearr), & + file=__FILE__, line=__LINE__) + endif + else + rearranger = PIO_REARR_BOX endif - !--- initialize ice_pio_subsystem nprocs = get_num_procs() - istride = 4 - basetask = min(1,nprocs-1) - numiotasks = max((nprocs-basetask)/istride,1) -!--tcraig this should work better but it causes pio2.4.4 to fail for reasons unknown -! numiotasks = 1 + (nprocs-basetask-1)/istride - rearranger = PIO_REARR_BOX - if (my_task == master_task) then + lstride = 4 + lroot = min(1,nprocs-1) +! Adjustments for PIO2 iotask issue, https://github.com/NCAR/ParallelIO/issues/1986 +! liotasks = max(1,(nprocs-lroot)/lstride) ! very conservative + liotasks = max(1,nprocs/lstride - lroot/lstride) ! less conservative (note integer math) +! liotasks = 1 + (nprocs-lroot-1)/lstride ! optimal + + if (present(iotasks)) then + if (iotasks /= -99) liotasks=iotasks + endif + if (present(root)) then + if (root /= -99) lroot=root + endif + if (present(stride)) then + if (stride /= -99) lstride=stride + endif + + if (liotasks < 1 .or. lroot < 0 .or. lstride < 1) then + call abort_ice(subname//' ERROR: iotasks, root, stride incorrect ', & + file=__FILE__, line=__LINE__) + endif + + ! adjust to fit in nprocs, preserve root and stride as much as possible + lroot = min(lroot,nprocs-1) ! lroot <= nprocs-1 +! Adjustments for PIO2 iotask issue, https://github.com/NCAR/ParallelIO/issues/1986 +! liotasks = max(1,min(liotasks, (nprocs-lroot)/lstride)) ! very conservative + liotasks = max(1,min(liotasks,nprocs/lstride - lroot/lstride)) ! less conservative (note integer math) +! liotasks = max(1,min(liotasks, 1 + (nprocs-lroot-1)/lstride)) ! optimal + + !--- initialize ice_pio_subsystem + + if (ldebug .and. my_task == master_task) then write(nu_diag,*) subname,' nprocs = ',nprocs - write(nu_diag,*) subname,' istride = ',istride - write(nu_diag,*) subname,' basetask = ',basetask - write(nu_diag,*) subname,' numiotasks = ',numiotasks write(nu_diag,*) subname,' pio_iotype = ',pio_iotype + write(nu_diag,*) subname,' iotasks = ',liotasks + write(nu_diag,*) subname,' baseroot = ',lroot + write(nu_diag,*) subname,' stride = ',lstride + write(nu_diag,*) subname,' nmode = ',nmode0 end if - call pio_init(my_task, MPI_COMM_ICE, numiotasks, master_task, istride, & - rearranger, ice_pio_subsystem, base=basetask) + call pio_init(my_task, MPI_COMM_ICE, liotasks, master_task, lstride, & + rearranger, ice_pio_subsystem, base=lroot) call pio_seterrorhandling(ice_pio_subsystem, PIO_RETURN_ERROR) - !--- initialize rearranger options - !pio_rearr_opt_comm_type = integer (PIO_REARR_COMM_[P2P,COLL]) - !pio_rearr_opt_fcd = integer, flow control (PIO_REARR_COMM_FC_[2D_ENABLE,1D_COMP2IO,1D_IO2COMP,2D_DISABLE]) - !pio_rearr_opt_c2i_enable_hs = logical - !pio_rearr_opt_c2i_enable_isend = logical - !pio_rearr_opt_c2i_max_pend_req = integer - !pio_rearr_opt_i2c_enable_hs = logical - !pio_rearr_opt_i2c_enable_isend = logical - !pio_rearr_opt_c2i_max_pend_req = integer - !ret = pio_set_rearr_opts(ice_pio_subsystem, pio_rearr_opt_comm_type,& - ! pio_rearr_opt_fcd,& - ! pio_rearr_opt_c2i_enable_hs, pio_rearr_opt_c2i_enable_isend,& - ! pio_rearr_opt_c2i_max_pend_req,& - ! pio_rearr_opt_i2c_enable_hs, pio_rearr_opt_i2c_enable_isend,& - ! pio_rearr_opt_i2c_max_pend_req) - !if(ret /= PIO_NOERR) then - ! call abort_ice(subname//'ERROR: aborting in pio_set_rearr_opts') - !end if - #endif if (present(mode) .and. present(filename) .and. present(File)) then if (trim(mode) == 'write') then - lclobber = .false. - if (present(clobber)) lclobber=clobber - lcdf64 = .false. - if (present(cdf64)) lcdf64=cdf64 + lclobber = .false. + if (present(clobber)) then + lclobber=clobber + endif if (File%fh<0) then ! filename not open inquire(file=trim(filename),exist=exists) if (exists) then if (lclobber) then - nmode = pio_clobber - if (lcdf64) nmode = ior(nmode,PIO_64BIT_OFFSET) + nmode = ior(PIO_CLOBBER,nmode0) status = pio_createfile(ice_pio_subsystem, File, pio_iotype, trim(filename), nmode) - call ice_pio_check(status, subname//' ERROR: Failed to create file '//trim(filename), & + call ice_pio_check(status, subname//' ERROR: Failed to overwrite file '//trim(filename), & file=__FILE__,line=__LINE__) if (my_task == master_task) then write(nu_diag,*) subname,' create file ',trim(filename) @@ -168,8 +217,7 @@ subroutine ice_pio_init(mode, filename, File, clobber, cdf64, iotype) end if endif else - nmode = pio_noclobber - if (lcdf64) nmode = ior(nmode,PIO_64BIT_OFFSET) + nmode = ior(PIO_NOCLOBBER,nmode0) status = pio_createfile(ice_pio_subsystem, File, pio_iotype, trim(filename), nmode) call ice_pio_check( status, subname//' ERROR: Failed to create file '//trim(filename), & file=__FILE__,line=__LINE__) diff --git a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_restart.F90 b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_restart.F90 index e55acc434..fdb9330d2 100644 --- a/cicecore/cicedyn/infrastructure/io/io_pio2/ice_restart.F90 +++ b/cicecore/cicedyn/infrastructure/io/io_pio2/ice_restart.F90 @@ -10,10 +10,7 @@ module ice_restart use ice_exit, only: abort_ice use ice_fileunits, only: nu_diag, nu_restart, nu_rst_pointer use ice_kinds_mod - use ice_restart_shared, only: & - restart, restart_ext, restart_dir, restart_file, pointer_file, & - runid, runtype, use_restart_time, restart_format, lcdf64, lenstr, & - restart_coszen + use ice_restart_shared use ice_pio use pio use icepack_intfc, only: icepack_warnings_flush, icepack_warnings_aborted @@ -32,6 +29,8 @@ module ice_restart type(io_desc_t) :: iodesc2d type(io_desc_t) :: iodesc3d_ncat + integer (kind=int_kind) :: dimid_ni, dimid_nj + !======================================================================= contains @@ -55,7 +54,9 @@ subroutine init_restart_read(ice_ic) character(len=char_len_long) :: & filename, filename0 - integer (kind=int_kind) :: status, iotype + integer (kind=int_kind) :: status + + logical (kind=log_kind), save :: first_call = .true. character(len=*), parameter :: subname = '(init_restart_read)' @@ -76,15 +77,21 @@ subroutine init_restart_read(ice_ic) write(nu_diag,*) 'Using restart dump=', trim(filename) end if - iotype = PIO_IOTYPE_NETCDF - if (restart_format == 'pio_pnetcdf') iotype = PIO_IOTYPE_PNETCDF File%fh=-1 - call ice_pio_init(mode='read', filename=trim(filename), File=File, iotype=iotype) +! tcraig, including fformat here causes some problems when restart_format=hdf5 +! and reading non hdf5 files with spack built PIO. Excluding the fformat +! argument here defaults the PIO format to cdf1 which then reads +! any netcdf format file fine. + call ice_pio_init(mode='read', filename=trim(filename), File=File, & +! fformat=trim(restart_format), rearr=trim(restart_rearranger), & + rearr=trim(restart_rearranger), & + iotasks=restart_iotasks, root=restart_root, stride=restart_stride, & + debug=first_call) call pio_seterrorhandling(File, PIO_RETURN_ERROR) call ice_pio_initdecomp(iodesc=iodesc2d, precision=8) - call ice_pio_initdecomp(ndim3=ncat , iodesc=iodesc3d_ncat,remap=.true., precision=8) + call ice_pio_initdecomp(ndim3=ncat, iodesc=iodesc3d_ncat, remap=.true., precision=8) if (use_restart_time) then ! for backwards compatibility, check nyr, month, and sec as well @@ -133,6 +140,8 @@ subroutine init_restart_read(ice_ic) npt = npt - istep0 endif + first_call = .false. + end subroutine init_restart_read !======================================================================= @@ -172,17 +181,16 @@ subroutine init_restart_write(filename_spec) character(len=char_len_long) :: filename integer (kind=int_kind) :: & - dimid_ni, dimid_nj, dimid_ncat, & - dimid_nilyr, dimid_nslyr, dimid_naero + dimid_ncat, dimid_nilyr, dimid_nslyr, dimid_naero integer (kind=int_kind), allocatable :: dims(:) - integer (kind=int_kind) :: iotype - integer (kind=int_kind) :: k, n ! loop index character (len=3) :: nchar, ncharb + logical (kind=log_kind), save :: first_call = .true. + character(len=*), parameter :: subname = '(init_restart_write)' call icepack_query_tracer_sizes(nbtrcr_out=nbtrcr) @@ -222,11 +230,11 @@ subroutine init_restart_write(filename_spec) close(nu_rst_pointer) endif - iotype = PIO_IOTYPE_NETCDF - if (restart_format == 'pio_pnetcdf') iotype = PIO_IOTYPE_PNETCDF File%fh=-1 call ice_pio_init(mode='write',filename=trim(filename), File=File, & - clobber=.true., cdf64=lcdf64, iotype=iotype) + clobber=.true., fformat=trim(restart_format), rearr=trim(restart_rearranger), & + iotasks=restart_iotasks, root=restart_root, stride=restart_stride, & + debug=first_call) call pio_seterrorhandling(File, PIO_RETURN_ERROR) @@ -674,6 +682,8 @@ subroutine init_restart_write(filename_spec) write(nu_diag,*) 'Writing ',filename(1:lenstr(filename)) endif + first_call = .false. + end subroutine init_restart_write !======================================================================= @@ -907,14 +917,44 @@ end subroutine final_restart subroutine define_rest_field(File, vname, dims) +#ifndef USE_PIO1 + use netcdf, only: NF90_CHUNKED + use pio_nf, only: pio_def_var_chunking !PIO <2.6.0 was missing this in the pio module +#endif type(file_desc_t) , intent(in) :: File character (len=*) , intent(in) :: vname integer (kind=int_kind), intent(in) :: dims(:) + integer (kind=int_kind) :: chunks(size(dims)), i, status + character(len=*), parameter :: subname = '(define_rest_field)' - call ice_pio_check(pio_def_var(File,trim(vname),pio_double,dims,vardesc), & - subname//' ERROR: def_var '//trim(vname),file=__FILE__,line=__LINE__) + + status = pio_def_var(File,trim(vname),pio_double,dims,vardesc) + call ice_pio_check(status, & + subname//' ERROR defining restart field '//trim(vname)) + +#ifndef USE_PIO1 + if (restart_format=='hdf5' .and. restart_deflate/=0) then + status = pio_def_var_deflate(File, vardesc, shuffle=0, deflate=1, deflate_level=restart_deflate) + call ice_pio_check(status, & + subname//' ERROR: deflating restart field '//trim(vname),file=__FILE__,line=__LINE__) + endif + + if (restart_format=='hdf5' .and. size(dims)>1) then + if (dims(1)==dimid_ni .and. dims(2)==dimid_nj) then + chunks(1)=restart_chunksize(1) + chunks(2)=restart_chunksize(2) + do i = 3, size(dims) + chunks(i) = 0 + enddo + + status = pio_def_var_chunking(File, vardesc, NF90_CHUNKED, chunks) + call ice_pio_check(status, subname//' ERROR: chunking restart field '//trim(vname),& + file=__FILE__,line=__LINE__) + endif + endif +#endif end subroutine define_rest_field diff --git a/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90 b/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90 index 5dec8a942..efadabbda 100644 --- a/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90 +++ b/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90 @@ -30,8 +30,9 @@ module ice_comp_nuopc use ice_kinds_mod , only : dbl_kind, int_kind, char_len, char_len_long use ice_fileunits , only : nu_diag, nu_diag_set, inst_index, inst_name use ice_fileunits , only : inst_suffix, release_all_fileunits, flush_fileunit - use ice_restart_shared , only : runid, runtype, restart, use_restart_time, restart_dir, restart_file + use ice_restart_shared , only : runid, runtype, restart, use_restart_time, restart_dir, restart_file, restart_format, restart_chunksize use ice_history , only : accum_hist + use ice_history_shared , only : history_format, history_chunksize use ice_exit , only : abort_ice use icepack_intfc , only : icepack_warnings_flush, icepack_warnings_aborted use icepack_intfc , only : icepack_init_orbit, icepack_init_parameters, icepack_query_orbit @@ -645,6 +646,36 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc) call abort_ice(trim(errmsg)) endif + ! Netcdf output created by PIO + call NUOPC_CompAttributeGet(gcomp, name="pio_typename", value=cvalue, & + isPresent=isPresent, isSet=isSet, rc=rc) + if (chkerr(rc,__LINE__,u_FILE_u)) return + if (isPresent .and. isSet) then + if (trim(history_format)/='cdf1' .and. mastertask) then + write(nu_diag,*) trim(subname)//history_format//'WARNING: history_format from cice_namelist ignored' + write(nu_diag,*) trim(subname)//'WARNING: using '//trim(cvalue)//' from ICE_modelio' + endif + if (trim(restart_format)/='cdf1' .and. mastertask) then + write(nu_diag,*) trim(subname)//restart_format//'WARNING: restart_format from cice_namelist ignored' + write(nu_diag,*) trim(subname)//'WARNING: using '//trim(cvalue)//' from ICE_modelio' + endif + + ! The only reason to set these is to detect in ice_history_write if the chunk/deflate settings are ok. + select case (trim(cvalue)) + case ('netcdf4p') + history_format='hdf5' + restart_format='hdf5' + case ('netcdf4c') + if (mastertask) write(nu_diag,*) trim(subname)//'WARNING: pio_typename = netcdf4c is superseded, use netcdf4p' + history_format='hdf5' + restart_format='hdf5' + case default !pio_typename=netcdf or pnetcdf + ! do nothing + end select + else + if(mastertask) write(nu_diag,*) trim(subname)//'WARNING: pio_typename from driver needs to be set for netcdf output to work' + end if + #else ! Read the cice namelist as part of the call to cice_init1 diff --git a/cicecore/drivers/standalone/cice/CICE_InitMod.F90 b/cicecore/drivers/standalone/cice/CICE_InitMod.F90 index a48bdda30..194293118 100644 --- a/cicecore/drivers/standalone/cice/CICE_InitMod.F90 +++ b/cicecore/drivers/standalone/cice/CICE_InitMod.F90 @@ -66,7 +66,7 @@ subroutine cice_init floe_binwidth, c_fsd_range use ice_state, only: alloc_state use ice_flux_bgc, only: alloc_flux_bgc - use ice_calendar, only: dt, dt_dyn, write_ic, & + use ice_calendar, only: dt, write_ic, & init_calendar, advance_timestep, calc_timesteps use ice_communicate, only: init_communicate, my_task, master_task use ice_diagnostics, only: init_diags diff --git a/cicecore/shared/ice_restart_shared.F90 b/cicecore/shared/ice_restart_shared.F90 index 7c178fec0..c022d77ba 100644 --- a/cicecore/shared/ice_restart_shared.F90 +++ b/cicecore/shared/ice_restart_shared.F90 @@ -26,9 +26,16 @@ module ice_restart_shared pointer_file ! input pointer file for restarts character (len=char_len), public :: & - restart_format ! format of restart files 'nc' + restart_format , & ! format of restart files 'nc' + restart_rearranger ! restart file rearranger, box or subset for pio + + integer (kind=int_kind), public :: & + restart_iotasks , & ! iotasks, root, stride defines io pes for pio + restart_root , & ! iotasks, root, stride defines io pes for pio + restart_stride , & ! iotasks, root, stride defines io pes for pio + restart_deflate , & ! compression level for hdf5/netcdf4 + restart_chunksize(2) ! chunksize for hdf5/netcdf4 - logical (kind=log_kind), public :: lcdf64 !======================================================================= diff --git a/configuration/scripts/ice_in b/configuration/scripts/ice_in index e33d16c18..103c56d2a 100644 --- a/configuration/scripts/ice_in +++ b/configuration/scripts/ice_in @@ -15,6 +15,12 @@ restart_ext = .false. use_restart_time = .false. restart_format = 'default' + restart_rearranger = 'default' + restart_iotasks = -99 + restart_root = -99 + restart_stride = -99 + restart_deflate = 0 + restart_chunksize = 0, 0 lcdf64 = .false. numin = 21 numax = 89 @@ -54,6 +60,12 @@ history_file = 'iceh' history_precision = 4 history_format = 'default' + history_rearranger = 'default' + history_iotasks = -99 + history_root = -99 + history_stride = -99 + history_deflate = 0 + history_chunksize = 0, 0 hist_time_axis = 'end' write_ic = .true. incond_dir = './history/' diff --git a/configuration/scripts/machines/env.derecho_cray b/configuration/scripts/machines/env.derecho_cray index 5294fbe95..47cebd5cb 100644 --- a/configuration/scripts/machines/env.derecho_cray +++ b/configuration/scripts/machines/env.derecho_cray @@ -23,6 +23,8 @@ module load cray-libsci/23.02.1.1 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/machines/env.derecho_gnu b/configuration/scripts/machines/env.derecho_gnu index 0f2d2ec87..5c4ca46f0 100644 --- a/configuration/scripts/machines/env.derecho_gnu +++ b/configuration/scripts/machines/env.derecho_gnu @@ -23,6 +23,8 @@ module load cray-libsci/23.02.1.1 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/machines/env.derecho_intel b/configuration/scripts/machines/env.derecho_intel index 7c822c923..63626dc33 100644 --- a/configuration/scripts/machines/env.derecho_intel +++ b/configuration/scripts/machines/env.derecho_intel @@ -23,6 +23,8 @@ module load cray-libsci/23.02.1.1 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/machines/env.derecho_intelclassic b/configuration/scripts/machines/env.derecho_intelclassic index 964f5e8bb..8d3639a5e 100644 --- a/configuration/scripts/machines/env.derecho_intelclassic +++ b/configuration/scripts/machines/env.derecho_intelclassic @@ -23,6 +23,8 @@ module load netcdf/4.9.2 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/machines/env.derecho_inteloneapi b/configuration/scripts/machines/env.derecho_inteloneapi index 700830525..8f3911036 100644 --- a/configuration/scripts/machines/env.derecho_inteloneapi +++ b/configuration/scripts/machines/env.derecho_inteloneapi @@ -23,6 +23,8 @@ module load cray-libsci/23.02.1.1 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/machines/env.derecho_nvhpc b/configuration/scripts/machines/env.derecho_nvhpc index f6bdf1138..34342769c 100644 --- a/configuration/scripts/machines/env.derecho_nvhpc +++ b/configuration/scripts/machines/env.derecho_nvhpc @@ -23,6 +23,8 @@ module load cray-libsci/23.02.1.1 if ($?ICE_IOTYPE) then if ($ICE_IOTYPE =~ pio*) then + module unload netcdf + module load netcdf-mpi/4.9.2 module load parallel-netcdf/1.12.3 if ($ICE_IOTYPE == "pio1") then module load parallelio/1.10.1 diff --git a/configuration/scripts/options/set_env.iopio1p b/configuration/scripts/options/set_env.iopio1p deleted file mode 100644 index 1a92353ce..000000000 --- a/configuration/scripts/options/set_env.iopio1p +++ /dev/null @@ -1,2 +0,0 @@ -setenv ICE_IOTYPE pio1 -setenv ICE_CPPDEFS -DUSE_PIO1 diff --git a/configuration/scripts/options/set_env.iopio2p b/configuration/scripts/options/set_env.iopio2p deleted file mode 100644 index 415005ac4..000000000 --- a/configuration/scripts/options/set_env.iopio2p +++ /dev/null @@ -1 +0,0 @@ -setenv ICE_IOTYPE pio2 diff --git a/configuration/scripts/options/set_nml.iobinary b/configuration/scripts/options/set_nml.iobinary index 7019acf0b..80ea92d61 100644 --- a/configuration/scripts/options/set_nml.iobinary +++ b/configuration/scripts/options/set_nml.iobinary @@ -1 +1,3 @@ ice_ic = 'internal' +history_format = 'binary' +restart_format = 'binary' diff --git a/configuration/scripts/options/set_nml.iocdf1 b/configuration/scripts/options/set_nml.iocdf1 new file mode 100644 index 000000000..ed9f65b68 --- /dev/null +++ b/configuration/scripts/options/set_nml.iocdf1 @@ -0,0 +1,2 @@ +restart_format = 'cdf1' +history_format = 'cdf1' diff --git a/configuration/scripts/options/set_nml.iocdf2 b/configuration/scripts/options/set_nml.iocdf2 new file mode 100644 index 000000000..ce10ae984 --- /dev/null +++ b/configuration/scripts/options/set_nml.iocdf2 @@ -0,0 +1,2 @@ +restart_format = 'cdf2' +history_format = 'cdf2' diff --git a/configuration/scripts/options/set_nml.iocdf5 b/configuration/scripts/options/set_nml.iocdf5 new file mode 100644 index 000000000..5081a8ac4 --- /dev/null +++ b/configuration/scripts/options/set_nml.iocdf5 @@ -0,0 +1,2 @@ +restart_format = 'cdf5' +history_format = 'cdf5' diff --git a/configuration/scripts/options/set_nml.iohdf5 b/configuration/scripts/options/set_nml.iohdf5 new file mode 100644 index 000000000..605a27938 --- /dev/null +++ b/configuration/scripts/options/set_nml.iohdf5 @@ -0,0 +1,2 @@ +restart_format = 'hdf5' +history_format = 'hdf5' diff --git a/configuration/scripts/options/set_nml.iohdf5opts b/configuration/scripts/options/set_nml.iohdf5opts new file mode 100644 index 000000000..6c780c169 --- /dev/null +++ b/configuration/scripts/options/set_nml.iohdf5opts @@ -0,0 +1,4 @@ +history_deflate = 6 +history_chunksize = 50,58 +restart_deflate = 8 +restart_chunksize = 50,58 diff --git a/configuration/scripts/options/set_nml.iopio1 b/configuration/scripts/options/set_nml.iopio1 deleted file mode 100644 index 655f2c96b..000000000 --- a/configuration/scripts/options/set_nml.iopio1 +++ /dev/null @@ -1,2 +0,0 @@ -restart_format = 'pio_netcdf' -history_format = 'pio_netcdf' diff --git a/configuration/scripts/options/set_nml.iopio1p b/configuration/scripts/options/set_nml.iopio1p deleted file mode 100644 index 83c422403..000000000 --- a/configuration/scripts/options/set_nml.iopio1p +++ /dev/null @@ -1,2 +0,0 @@ -restart_format = 'pio_pnetcdf' -history_format = 'pio_pnetcdf' diff --git a/configuration/scripts/options/set_nml.iopio2 b/configuration/scripts/options/set_nml.iopio2 deleted file mode 100644 index 655f2c96b..000000000 --- a/configuration/scripts/options/set_nml.iopio2 +++ /dev/null @@ -1,2 +0,0 @@ -restart_format = 'pio_netcdf' -history_format = 'pio_netcdf' diff --git a/configuration/scripts/options/set_nml.iopio2p b/configuration/scripts/options/set_nml.iopio2p deleted file mode 100644 index e4cce54af..000000000 --- a/configuration/scripts/options/set_nml.iopio2p +++ /dev/null @@ -1,2 +0,0 @@ -restart_format = 'pio_pnetcdf' -history_format = 'pio_netcdf' diff --git a/configuration/scripts/options/set_nml.iopioopts b/configuration/scripts/options/set_nml.iopioopts new file mode 100644 index 000000000..63aaeefcf --- /dev/null +++ b/configuration/scripts/options/set_nml.iopioopts @@ -0,0 +1,10 @@ +history_format = 'cdf2' +history_rearranger = 'subset' +history_iotasks = 1024 +history_root = 0 +history_stride = 2 +restart_format = 'pnetcdf5' +restart_rearranger = 'subset' +restart_iotasks = 1024 +restart_root = 1024 +restart_stride = 8 diff --git a/configuration/scripts/options/set_nml.iopnetcdf1 b/configuration/scripts/options/set_nml.iopnetcdf1 new file mode 100644 index 000000000..9346ed637 --- /dev/null +++ b/configuration/scripts/options/set_nml.iopnetcdf1 @@ -0,0 +1,2 @@ +restart_format = 'pnetcdf1' +history_format = 'pnetcdf1' diff --git a/configuration/scripts/options/set_nml.iopnetcdf2 b/configuration/scripts/options/set_nml.iopnetcdf2 new file mode 100644 index 000000000..27dd6f51c --- /dev/null +++ b/configuration/scripts/options/set_nml.iopnetcdf2 @@ -0,0 +1,2 @@ +restart_format = 'pnetcdf2' +history_format = 'pnetcdf2' diff --git a/configuration/scripts/options/set_nml.iopnetcdf5 b/configuration/scripts/options/set_nml.iopnetcdf5 new file mode 100644 index 000000000..3c95890d9 --- /dev/null +++ b/configuration/scripts/options/set_nml.iopnetcdf5 @@ -0,0 +1,2 @@ +restart_format = 'pnetcdf5' +history_format = 'pnetcdf5' diff --git a/configuration/scripts/tests/io_suite.ts b/configuration/scripts/tests/io_suite.ts index 84d064f32..e5e7feee6 100644 --- a/configuration/scripts/tests/io_suite.ts +++ b/configuration/scripts/tests/io_suite.ts @@ -15,73 +15,47 @@ restart gx3 14x2 gx3ncarbulk,isotope,histall,iobinary,precision restart gx3 16x2 gx3ncarbulk,fsd12,histall,iobinary restart gx3 8x4 gx3ncarbulk,debug,histall,iobinary,precision8,histinst -restart gx3 32x1 debug,histall,ionetcdf -restart gx3 15x2 alt01,histall,ionetcdf,precision8,cdf64 -restart gx3 15x2 alt02,histall,ionetcdf -restart gx3 24x1 alt03,histall,ionetcdf,precision8 -restart gx3 8x4 alt04,histall,ionetcdf,cdf64 -restart gx3 8x4 alt05,histall,ionetcdf,precision8,cdf64 -restart gx3 16x2 alt06,histall,ionetcdf -restart gx3 16x2 alt07,histall,ionetcdf -restart gx3 30x1 bgczm,histall,ionetcdf -restart gx3 15x2 bgcskl,histall,ionetcdf,precision8 -restart gx3 31x1 isotope,histall,ionetcdf,cdf64 -restart gx3 14x2 fsd12,histall,ionetcdf,precision8 -restart gx3 32x1 debug,histall,ionetcdf,histinst +restart gx3 32x1 debug,histall,ionetcdf,iocdf1,precision8 +restart gx3 15x2 alt01,histall,ionetcdf,iocdf2,precision8 +restart gx3 15x2 alt02,histall,ionetcdf,iocdf5 +restart gx3 24x1 alt03,histall,ionetcdf,iohdf5,iohdf5opts +restart gx3 8x4 alt04,histall,ionetcdf,iocdf1 +restart gx3 8x4 alt05,histall,ionetcdf,iocdf2 +restart gx3 16x2 alt06,histall,ionetcdf,iocdf5,precision8 +restart gx3 16x2 alt07,histall,ionetcdf,iohdf5,precision8 +restart gx3 30x1 bgczm,histall,ionetcdf,iocdf1 +restart gx3 15x2 bgcskl,histall,ionetcdf,iocdf2,precision8 +restart gx3 31x1 isotope,histall,ionetcdf,iocdf5,precision8 +restart gx3 14x2 fsd12,histall,ionetcdf,iohdf5 +restart gx3 32x1 debug,histall,ionetcdf,iohdf5,histinst -restart gx3 16x2 debug,histall,iopio1,precision8,cdf64 -restart gx3 14x2 alt01,histall,iopio1,cdf64 -restart gx3 32x1 alt02,histall,iopio1,precision8 -restart gx3 24x1 alt03,histall,iopio1 -restart gx3 8x4 alt04,histall,iopio1,precision8,cdf64 -restart gx3 8x4 alt05,histall,iopio1,cdf64 -restart gx3 32x1 alt06,histall,iopio1,precision8 -restart gx3 32x1 alt07,histall,iopio1,precision8 -restart gx3 16x2 bgczm,histall,iopio1,precision8 -restart gx3 30x1 bgcskl,histall,iopio1 -restart gx3 8x4 isotope,histall,iopio1,precision8,cdf64 -restart gx3 12x2 fsd12,histall,iopio1,cdf64 -restart gx3 16x2 debug,histall,iopio1,precision8,cdf64,histinst +restart gx3 16x2x100x2x4 histall,iopio1,iopioopts +restart gx3 16x2 debug,histall,iopio1,iocdf2 +restart gx3 14x2 alt01,histall,iopio1,iocdf5 +restart gx3 32x1 alt02,histall,iopio1,iohdf5 +restart gx3 24x1 alt03,histall,iopio1,iopnetcdf1,precision8 +restart gx3 8x4 alt04,histall,iopio1,iopnetcdf2,precision8 +restart gx3 8x4 alt05,histall,iopio1,iopnetcdf5,precision8 +restart gx3 32x1 alt06,histall,iopio1,iocdf1 +restart gx3 32x1 alt07,histall,iopio1,iocdf2,precision8 +restart gx3 16x2 bgczm,histall,iopio1,iocdf5,precision8 +restart gx3 30x1 bgcskl,histall,iopio1,iohdf5,precision8 +restart gx3 8x4 isotope,histall,iopio1,iopnetcdf1 +restart gx3 12x2 fsd12,histall,iopio1,iopnetcdf2 +restart gx3 16x2 debug,histall,iopio1,iopnetcdf5,histinst -restart gx3 16x2 debug,histall,iopio2 -restart gx3 14x2 alt01,histall,iopio2,precision8,cdf64 -restart gx3 32x1 alt02,histall,iopio2,cdf64 -restart gx3 24x1 alt03,histall,iopio2,precision8 -restart gx3 8x4 alt04,histall,iopio2 -restart gx3 8x4 alt05,histall,iopio2,precision8,cdf64 -restart gx3 16x2 alt06,histall,iopio2,cdf64 -restart gx3 16x2 alt07,histall,iopio2,cdf64 -restart gx3 16x2 bgczm,histall,iopio2,cdf64 -restart gx3 30x1 bgcskl,histall,iopio2,precision8 -restart gx3 8x4 isotope,histall,iopio2 -restart gx3 12x2 fsd12,histall,iopio2,precision8,cdf64 -restart gx3 16x2 debug,histall,iopio2,histinst - -restart gx3 16x2 debug,histall,iopio1p,precision8 -restart gx3 14x2 alt01,histall,iopio1p -restart gx3 32x1 alt02,histall,iopio1p,precision8,cdf64 -restart gx3 24x1 alt03,histall,iopio1p,cdf64 -restart gx3 8x4 alt04,histall,iopio1p,precision8 -restart gx3 8x4 alt05,histall,iopio1p -restart gx3 6x4 alt06,histall,iopio1p,precision8,cdf64 -restart gx3 6x4 alt07,histall,iopio1p,precision8,cdf64 -restart gx3 16x2 bgczm,histall,iopio1p,precision8,cdf64 -restart gx3 30x1 bgcskl,histall,iopio1p,cdf64 -restart gx3 8x4 isotope,histall,iopio1p,precision8 -restart gx3 12x2 fsd12,histall,iopio1p -restart gx3 16x2 debug,histall,iopio1p,precision8,histinst - -restart gx3 16x2 debug,histall,iopio2p,cdf64 -restart gx3 14x2 alt01,histall,iopio2p,precision8 -restart gx3 32x1 alt02,histall,iopio2p -restart gx3 24x1 alt03,histall,iopio2p,precision8,cdf64 -restart gx3 8x4 alt04,histall,iopio2p,cdf64 -restart gx3 8x4 alt05,histall,iopio2p,precision8 -restart gx3 24x1 alt06,histall,iopio2p -restart gx3 24x1 alt07,histall,iopio2p -restart gx3 16x2 bgczm,histall,iopio2p -restart gx3 30x1 bgcskl,histall,iopio2p,precision8,cdf64 -restart gx3 8x4 isotope,histall,iopio2p,cdf64 -restart gx3 12x2 fsd12,histall,iopio2p,precision8 -restart gx3 16x2 debug,histall,iopio2p,cdf64,histinst +restart gx3 16x2x100x2x4 debug,histall,iopio2,iopioopts,run5day +restart gx3 16x2 debug,histall,iopio2,iopnetcdf1,precision8 +restart gx3 14x2 alt01,histall,iopio2,iopnetcdf2,precision8 +restart gx3 32x1 alt02,histall,iopio2,iopnetcdf5,precision8 +restart gx3 24x1 alt03,histall,iopio2,iocdf1 +restart gx3 8x4 alt04,histall,iopio2,iocdf2 +restart gx3 8x4 alt05,histall,iopio2,iocdf5 +restart gx3 16x2 alt06,histall,iopio2,iohdf5,iohdf5opts +restart gx3 16x2 alt07,histall,iopio2,iopnetcdf1 +restart gx3 16x2 bgczm,histall,iopio2,iopnetcdf2 +restart gx3 30x1 bgcskl,histall,iopio2,iopnetcdf5 +restart gx3 8x4 isotope,histall,iopio2,iohdf5,precision8 +restart gx3 12x2 fsd12,histall,iopio2,iocdf1,precision8 +restart gx3 16x2 debug,histall,iopio2,iocdf2,histinst,precision8 diff --git a/doc/source/cice_index.rst b/doc/source/cice_index.rst index 1249feb08..6b97d2b8f 100644 --- a/doc/source/cice_index.rst +++ b/doc/source/cice_index.rst @@ -320,10 +320,16 @@ section :ref:`tabnamelist`. "histfreq", "units of history output frequency: y, m, w, d or 1", "m,x,x,x,x" "histfreq_base", "reference date for history output, zero or init", "" "histfreq_n", "integer output frequency in histfreq units", "1,1,1,1,1" + "history_chunksize", "history chunksizes in x,y directions (_format='hdf5' only)", "0,0" + "history_deflate", "compression level for history (_format='hdf5' only)", "0" "history_dir", "path to history output files", "" "history_file", "history output file prefix", "" "history_format", "history file format", "" + "history_iotasks", "history output total number of tasks used", "" "history_precision", "history output precision: 4 or 8 byte", "4" + "history_rearranger", "history output io rearranger method", "" + "history_root", "history output io root task id", "" + "history_stride", "history output io task stride", "" "hist_time_axis", "history file time axis interval location: begin, middle, end", "end" "hist_suffix", "suffix to `history_file` in filename. x means no suffix", "x,x,x,x,x" "hm", "land/boundary mask, thickness (T-cell)", "" @@ -577,9 +583,15 @@ section :ref:`tabnamelist`. "restart", "if true, initialize ice state from file", "T" "restart_age", "if true, read age restart file", "" "restart_bgc", "if true, read bgc restart file", "" + "restart_chunksize", "restart chunksizes in x,y directions (_format='hdf5' only)", "0,0" + "restart_deflate", "compression level for restart (_format='hdf5' only)", "0" "restart_dir", "path to restart/dump files", "" "restart_file", "restart file prefix", "" "restart_format", "restart file format", "" + "restart_iotasks", "restart output total number of tasks used", "" + "restart_rearranger", "restart output io rearranger method", "" + "restart_root", "restart output io root task id", "" + "restart_stride", "restart output io task stride", "" "restart_[tracer]", "if true, read tracer restart file", "" "restart_ext", "if true, read/write halo cells in restart file", "" "restart_coszen", "if true, read/write coszen in restart file", "" diff --git a/doc/source/developer_guide/dg_infra.rst b/doc/source/developer_guide/dg_infra.rst index c38e2c16d..7b7fb907a 100644 --- a/doc/source/developer_guide/dg_infra.rst +++ b/doc/source/developer_guide/dg_infra.rst @@ -40,7 +40,7 @@ Time manager data is module data in **cicecore/shared/ice_calendar.F90**. Much data is public and operated on during the model timestepping. The model timestepping actually takes place in the **CICE_RunMod.F90** file which is part of the driver code. -The time manager was updated in early 2021. Additional information about the time manager can be found here, :ref:`timemanagerplus` +The time manager was updated in early 2021. Additional information about the time manager can be found here, :ref:`timemanagerplus`. @@ -82,3 +82,5 @@ is a parallel io library (https://github.com/NCAR/ParallelIO) that supports read binary and netcdf file through various interfaces including netcdf and pnetcdf. pio is generally more parallel in memory even when using serial netcdf than the standard gather/scatter methods, and it provides parallel read/write capabilities by optionally linking and using pnetcdf. + +There is additional IO information in :ref:`modelio`. diff --git a/doc/source/user_guide/ug_case_settings.rst b/doc/source/user_guide/ug_case_settings.rst index 7ba3f35ad..b8bde525d 100644 --- a/doc/source/user_guide/ug_case_settings.rst +++ b/doc/source/user_guide/ug_case_settings.rst @@ -38,7 +38,7 @@ can be found in :ref:`cicecpps`. The following CPPs are available. "NO_R16", "Converts real*16 to real*8. This could have adverse affects for certain algorithms including the lsum16 implementation associated with the ``bfbflag``" "NO_SNICARHC", "Does not compile hardcoded (HC) 5 band snicar tables tables needed by ``shortwave=dEdd_snicar_ad``. May reduce compile time." "USE_NETCDF", "Turns on netCDF code. This is normally on and is needed for released configurations. An older value, ncdf, is still supported." - "USE_PIO1", "Modifies pio code to be compatible with PIO1. By default, code is compatible with PIO2" + "USE_PIO1", "Modifies CICE PIO implementation to be compatible with PIO1. By default, code is compatible with PIO2" "","" "**Application Macros**", "" "CESMCOUPLED", "Turns on code changes for the CESM coupled application " @@ -84,11 +84,13 @@ can be modified as needed. "ICE_IOTYPE", "string", "I/O source code", "set by cice.setup" " ", "binary", "uses io_binary directory, no support for netCDF files" " ", "netcdf", "uses io_netCDF directory, supports netCDF files" - " ", "pio", "uses io_pio directory, supports netCDF and parallel netCDF thru PIO interfaces" + " ", "pio1", "uses io_pio directory with PIO1 library, supports netCDF and parallel netCDF thru PIO interfaces" + " ", "pio2", "uses io_pio directory with PIO2 library, supports netCDF and parallel netCDF thru PIO interfaces" "ICE_CLEANBUILD", "true, false", "automatically clean before building", "true" "ICE_CPPDEFS", "user defined preprocessor macros for build", "null" "ICE_QUIETMODE", "true, false", "reduce build output to the screen", "false" "ICE_GRID", "string (see below)", "grid", "set by cice.setup" + " ", "gbox12", "12x12 box", " " " ", "gbox80", "80x80 box", " " " ", "gbox128", "128x128 box", " " " ", "gbox180", "180x180 box", " " @@ -194,11 +196,28 @@ setup_nml "``histfreq_base``", "init", "history output frequency relative to year_init, month_init, day_init", "'zero','zero','zero','zero','zero'" "", "zero", "history output frequency relative to year-month-day of 0000-01-01", "" "``histfreq_n``", "integer array", "frequency history output is written with ``histfreq``", "1,1,1,1,1" + "``history_chunksize``", "integer array", "chunksizes (x,y) for history output (hdf5 only)", "0,0" + "``history_deflate``", "integer", "compression level (0 to 9) for history output (hdf5 only)", "0" "``history_dir``", "string", "path to history output directory", "'./'" "``history_file``", "string", "output file for history", "'iceh'" - "``history_format``", "``default``", "read/write history files in default format", "``default``" - "", "``pio_pnetcdf``", "read/write restart files with pnetcdf in pio", "" + "``history_format``", "``binary``", "write history files with binary format", "``cdf1``" + "", "``cdf1``", "write history files with netcdf cdf1 (netcdf3-classic) format", "" + "", "``cdf2``", "write history files with netcdf cdf2 (netcdf3-64bit-offset) format", "" + "", "``cdf5``", "write history files with netcdf cdf5 (netcdf3-64bit-data) format", "" + "", "``default``", "write history files in default format", "" + "", "``hdf5``", "write history files with netcdf hdf5 (netcdf4) format", "" + "", "``pio_pnetcdf``", "write history files with pnetcdf in PIO, deprecated", "" + "", "``pio_netcdf``", "write history files with netcdf in PIO, deprecated", "" + "", "``pnetcdf1``", "write history files with pnetcdf cdf1 (netcdf3-classic) format", "" + "", "``pnetcdf2``", "write history files with pnetcdf cdf2 (netcdf3-64bit-offset) format", "" + "", "``pnetcdf5``", "write history files with pnetcdf cdf5 (netcdf3-64bit-data) format", "" + "``history_iotasks``", "integer", "pe io tasks for history output with history_root and history_stride (PIO only), -99=internal default", "-99" "``history_precision``", "integer", "history file precision: 4 or 8 byte", "4" + "``history_rearranger``", "box", "box io rearranger option for history output (PIO only)", "default" + "", "default", "internal default io rearranger option for history output", "" + "", "subset", "subset io rearranger option for history output", "" + "``history_root``", "integer", "pe root task for history output with history_iotasks and history_stride (PIO only), -99=internal default", "-99" + "``history_stride``", "integer", "pe stride for history output with history_iotasks and history_root (PIO only), -99=internal default", "-99" "``hist_suffix``", "character array", "appended to history_file when not x", "``x,x,x,x,x``" "``hist_time_axis``","character","history file time axis interval location: begin, middle, end","end" "``ice_ic``", "``default``", "equal to internal", "``default``" @@ -209,7 +228,7 @@ setup_nml "``incond_file``", "string", "output file prefix for initial condition", "‘iceh_ic’" "``istep0``", "integer", "initial time step number", "0" "``latpnt``", "real", "latitude of (2) diagnostic points", "90.0,-65.0" - "``lcdf64``", "logical", "use 64-bit netCDF format", "``.false.``" + "``lcdf64``", "logical", "use 64-bit netCDF format, deprecated, see history_format, restart_format", "``.false.``" "``lonpnt``", "real", "longitude of (2) diagnostic points", "0.0,-45.0" "``memory_stats``", "logical", "turns on memory use diagnostics", "``.false.``" "``month_init``", "integer", "the initial month if not using restart", "1" @@ -227,11 +246,28 @@ setup_nml "``print_global``", "logical", "print global sums diagnostic data", "``.true.``" "``print_points``", "logical", "print diagnostic data for two grid points", "``.false.``" "``restart``", "logical", "exists but deprecated, now set internally based on other inputs", "" + "``restart_chunksize``", "integer array", "chunksizes (x,y) for restart output (hdf5 only)", "0,0" + "``restart_deflate``", "integer", "compression level (0 to 9) for restart output (hdf5 only)", "0" "``restart_dir``", "string", "path to restart directory", "'./'" "``restart_ext``", "logical", "read/write halo cells in restart files", "``.false.``" "``restart_file``", "string", "output file prefix for restart dump", "'iced'" - "``restart_format``", "``default``", "read/write restart file with default format", "``default``" - "", "``pio_pnetcdf``", "read/write restart files with pnetcdf in pio", "" + "``restart_format``", "``binary``", "write restart files with binary format", "``cdf1``" + "", "``cdf1``", "write restart files with netcdf cdf1 (netcdf3-classic) format", "" + "", "``cdf2``", "write restart files with netcdf cdf2 (netcdf3-64bit-offset) format", "" + "", "``cdf5``", "write restart files with netcdf cdf5 (netcdf3-64bit-data) format", "" + "", "``default``", "write restart files in default format", "" + "", "``hdf5``", "write restart files with netcdf hdf5 (netcdf4) format", "" + "", "``pio_pnetcdf``", "write restart files with pnetcdf in PIO, deprecated", "" + "", "``pio_netcdf``", "write restart files with netcdf in PIO, deprecated", "" + "", "``pnetcdf1``", "write restart files with pnetcdf cdf1 (netcdf3-classic) format", "" + "", "``pnetcdf2``", "write restart files with pnetcdf cdf2 (netcdf3-64bit-offset) format", "" + "", "``pnetcdf5``", "write restart files with pnetcdf cdf5 (netcdf3-64bit-data) format", "" + "``restart_iotasks``", "integer", "pe io tasks for restart output with restart_root and restart_stride (PIO only), -99=internal default", "-99" + "``restart_rearranger``", "box", "box io rearranger option for restart output (PIO only)", "default" + "", "default", "internal default io rearranger option for restart output", "" + "", "subset", "subset io rearranger option for restart output", "" + "``restart_root``", "integer", "pe root task for restart output with restart_iotasks and restart_stride (PIO only), -99=internal default", "-99" + "``restart_stride``", "integer", "pe stride for restart output with restart_iotasks and restart_root (PIO only), -99=internal default", "-99" "``runid``", "string", "label for run (currently CESM only)", "'unknown'" "``runtype``", "``continue``", "restart using ``pointer_file``", "``initial``" "", "``initial``", "start from ``ice_ic``", "" diff --git a/doc/source/user_guide/ug_implementation.rst b/doc/source/user_guide/ug_implementation.rst index a67fc3a58..c243616d2 100644 --- a/doc/source/user_guide/ug_implementation.rst +++ b/doc/source/user_guide/ug_implementation.rst @@ -747,7 +747,7 @@ characteristics. In the ‘sectcart’ case, the domain is divided into four (east-west,north-south) quarters and the loops are done over each, sequentially. The ``wghtfile`` decomposition drives the decomposition based on -weights provided in a weight file. That file should be a netcdf +weights provided in a weight file. That file should be a netCDF file with a double real field called ``wght`` containing the relative weight of each gridcell. :ref:`fig-distrbB` (b) and (c) show an example. The weights associated with each gridcell will be @@ -1136,11 +1136,89 @@ relaxation parameter ``arlx1i`` effectively sets the damping timescale in the problem, and ``brlx`` represents the effective subcycling :cite:`Bouillon13` (see Section :ref:`revp`). -~~~~~~~~~~~~ -Model output -~~~~~~~~~~~~ +.. _modelio: -There are a number of model output streams and formats. +~~~~~~~~~~~~~~~~~~~~~~~~ +Model Input and Output +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _iooverview: + +************* +IO Overview +************* + +CICE provides the ability to read and write binary unformatted or netCDF +data via a number of different methods. The IO implementation is specified +both at build-time (via selection of specific source code) and run-time (via namelist). +Three different IO packages are available in CICE under the directory +**cicecore/cicedyn/infrastructure/io**. Those are io_binary, io_netcdf, and +io_pio2, and those support IO thru binary, netCDF (https://www.unidata.ucar.edu/software/netcdf), +and PIO (https://github.com/NCAR/ParallelIO) interfaces respectively. +The io_pio2 directory supports both PIO1 and PIO2 and can write data thru the +netCDF or parallel netCDF (pnetCDF) interface. The netCDF history files are CF-compliant, and +header information for data contained in the netCDF files is displayed with +the command ``ncdump -h filename.nc``. To select the io source code, set ``ICE_IOTYPE`` +in **cice.settings** to ``binary``, ``netcdf``, ``pio1``, or ``pio2``. + +At run-time, more detailed IO settings are available. ``restart_format`` and +``history_format`` namelist options specify the method and format further. Valid options +are listed in :ref:`formats`. These options specify the format of new files created +by CICE. Existing files can be read in any format as long as it's consistent +with ``ICE_IOTYPE`` defined. Note that with ``ICE_IOTYPE = binary``, the format name +is actually ignored. The CICE netCDF output contains a global metadata attribute, ``io_flavor``, +that indicates the format chosen for the file. ``ncdump -k filename.nc`` also +provides information about the specific netCDF file format. +In general, the detailed format is not enforced for input files, so any netCDF format +can be read in CICE regardless of CICE namelist settings. + +.. _formats: + +.. table:: CICE IO formats + + +--------------+----------------------+-------------+---------------------+ + | **Namelist** | **Format** | **Written** | **Valid With** | + | **Option** | | **Thru** | **ICE_IOTYPE** | + +--------------+----------------------+-------------+---------------------+ + | binary | Fortran binary | fortran | binary | + +--------------+----------------------+-------------+---------------------+ + | cdf1 | netCDF3-classic | netCDF | netcdf, pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | cdf2 | netCDF3-64bit-offset | netCDF | netcdf, pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | cdf5 | netCDF3-64bit-data | netCDF | netcdf, pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | default | binary or cdf1, | varies | binary, netcdf, | + | | depends on ICE_IOTYPE| | pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | hdf5 | netCDF4 hdf5 | netCDF | netcdf, pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | pnetcdf1 | netCDF3-classic | pnetCDF | pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | pnetcdf2 | netCDF3-64bit-offset | pnetCDF | pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + | pnetcdf5 | netCDF3-64bit-data | pnetCDF | pio1, pio2 | + +--------------+----------------------+-------------+---------------------+ + +There are additional namelist options that affect PIO performance for both +restart and history output. [``history_,restart_``] +[``iotasks,root,stride``] +namelist options control the PIO processor/task usage and specify the total number of +IO tasks, the root IO task, and the IO task stride respectively. +``history_rearranger`` and ``restart_rearranger`` +define the PIO rearranger strategy. Finally, [``history_,restart_``] +[``deflate,chunksize``] provide +controls for hdf5 compression and chunking for the ``hdf5`` options +in both netCDF and PIO output. ``hdf5`` is written serially thru the +netCDF library and in parallel thru the PIO library in CICE. Additional +details about the netCDF and PIO settings and implementations can +found in (https://www.unidata.ucar.edu/software/netcdf) +and (https://github.com/NCAR/ParallelIO). + +netCDF requires CICE compilation with a netCDF library built externally. +PIO requires CICE compilation with a PIO and netCDF library built externally. +Both netCDF and PIO can be built with many options which may require additional libraries +such as MPI, hdf5, or pnetCDF. .. _history: @@ -1148,16 +1226,13 @@ There are a number of model output streams and formats. History files ************* -CICE provides history data in binary unformatted or netCDF formats via -separate implementations of binary, netcdf, and pio source code under the -directory **infrastructure/io**. ``ICE_IOTYPE`` defined in cice.settings -specifies the IO type and defines which source code directory is compiled. -At the present time, binary, netcdf, and PIO are exclusive formats -for history and restart files, and history and restart file must use the same -io package. The namelist variable ``history_format`` further refines the -format approach or style for some io packages. +CICE provides history data output in binary unformatted or netCDF formats via +separate implementations of binary, netCDF, and PIO interfaces as described +above. In addition, ``history_format`` as well as other history namelist +options control the specific file format as well as features related to +IO performance, see :ref:`iooverview`. -Model output data can be written as instantaneous or average data as specified +CICE Model history output data can be written as instantaneous or average data as specified by the ``hist_avg`` namelist array and is customizable by stream. Characters can be added to the ``history_filename`` to distinguish the streams. This can be changed by modifying ``hist_suffix`` to something other than "x". @@ -1169,12 +1244,7 @@ in **ice_in**. These settings for history files are set in the **setup_nml** section of **ice_in** (see :ref:`tabnamelist`). If ``history_file`` = ‘iceh’ then the filenames will have the form **iceh.[timeID].nc** or **iceh.[timeID].da**, -depending on the output file format chosen in **cice.settings** (set -``ICE_IOTYPE``). The netCDF history files are CF-compliant; header information for -data contained in the netCDF files is displayed with the command ``ncdump -h -filename.nc``. Parallel netCDF output is available using the PIO library; the -output file attribute ``io_flavor`` distinguishes output files written with PIO from -those written with standard netCDF. With binary files, a separate header +depending on the output file format chosen. With binary files, a separate header file is written with equivalent information. Standard fields are output according to settings in the **icefields\_nml** section of **ice\_in** (see :ref:`tabnamelist`). @@ -1404,18 +1474,16 @@ The timers use *MPI\_WTIME* for parallel runs and the F90 intrinsic Restart files ************* -CICE provides restart data in binary unformatted or netCDF formats via -separate implementations of binary, netcdf, and pio source code under the -directory **infrastructure/io**. ``ICE_IOTYPE`` defined in cice.settings -specifies the IO type and defines which source code directory is compiled. -At the present time, binary, netcdf, and PIO are exclusive formats -for history and restart files, and history and restart file must use the same -io package. The namelist variable ``restart_format`` further refines the -format approach or style for some io packages. +CICE reads and writes restart data in binary unformatted or netCDF formats via +separate implementations of binary, netCDF, and PIO interfaces as described +above. In addition, ``restart_format`` as well as other restart namelist +options control the specific file format as well as features related to +IO performance, see :ref:`iooverview`. The restart files created by CICE contain all of the variables needed for a full, exact restart. The filename begins with the character string -‘iced.’, and the restart dump frequency is given by the namelist +defined by the ``restart_file`` namelist input, and the restart dump frequency +is given by the namelist variables ``dumpfreq`` and ``dumpfreq_n`` relative to a reference date specified by ``dumpfreq_base``. Multiple restart frequencies are supported in the code with a similar mechanism to history streams. The pointer to the filename from