Technical documentation

Creating remapping weights

The first step in coupling models together is the generation of remapping weights files. The weights files are used by the coupler (OASIS3-MCT or Tango) at runtime to transfer the coupling fields.

ESMFRegridWeightGen

For the 1 and 0.25 degree ACCESS-OM configurations OASIS3-MCT/SCRIP has always been used to generate the remapping weights. This approach does not scale well. At 1 deg the generation step takes several minutes, at 0.25 deg it takes 5-6 hours. This approach is impractical for a 0.1 degree grid.

Fortunately ESMF can be used instead. It supports multiprocessing and outputs weights files in a format that OASIS can use during runtime. In effect we are just replacing the SCRIP part of OASIS with ESMF. Furthermore the latest versions also support higher order interpolation schemes.

A tool, make_remap_weights.py, is included within the ACCESS-OM2 repository that uses ESMF to create weights files.

At the time of writing it is necessary to use a development snapshot of ESMF since there is currently no release version that supports the 2nd order conservative remapping scheme. On raijin this is done with:

cd $ACCESS_OM_DIR/tools/contrib
./build_esmf_on_raijin.sh

Check that the new ESMF_RegridWeightGen executable has been built successfully:

ls $ACCESS_OM_DIR/tools/contrib/bin/ESMF_RegridWeightGen

Regenerate the weights:

cd ../
qsub make_remap_weights.sh

Comparison of OASIS3-MCT/SCRIP and ESMF weight files

This section shows some runtime and error comparisons between OASIS-MCT/SCRIP and ESMF. They pass a field from a CORE2 grid to a MOM 1 degree grid and calculate the conservativeness and runtime of the remapping.

The tests on the 1 degree grid show that EMSF is more conservative than OASIS3-MCT/SCRIP:

Remapper	Relative Error	Runtime (s)
OASIS-MCT/SCRIP	9.5e-10	118.5
ESMF	3.4e-16	23.2

From the CORE2 to a MOM 0.1 degree grid:

Remapper	Relative Error	Runtime (s)
OASIS-MCT/SCRIP	-	-
ESMF	1.6e-14	169.7

There are no numbers for OASIS on the 0.1 degree grid, this did not complete after several hours.

The runtimes give the time it takes to build the weights file and remap a single field. They are rough and unoptimised, for example, the OASIS remapping is done in Fortran, while the ESMF one uses Python.

The OASIS3-MCT/SCRIP error looks high. It might be worth further investigation if this remapping approach is to be used. For the 0.1 degree configuration we are restricted to ESMF in any case for performance reasons.

Test status

Remapping weight creation and remapping error are tested by ARCCSS Jenkins

Summary

This section introduced new Python tools and packages that can:

• Generate weight files using the ESMF_RegridWeightGen tool. Using this overcomes performance limitation with OASIS3-MCT/SCRIP and makes it possible to easily build weights files for high resolution grids.
• Compare how well ESMF and SCRIP conserve fields passed between atmosphere and ocean grids. This shows that there is no downside to using ESMF over SCRIP.

Creating OASIS coupling fields restarts

This section describes how to create the initial coupling field restarts.

Scaling and Performance

Typical resource requirements are:

Model	Horizontal grid	Model domain	Baroclinic timestep (s)	MOM cores	CICE cores	Walltime (hr/yr)	CPU (hr/yr)	Memory (Gb)
ACCESS-OM2	1°	360×300×50	5400	216	24	0.38	118	83
ACCESS-OM2-025	0.25°	1440×1080×50	1350-1800	1455	361	2.6	4,700	522
ACCESS-OM2-01	0.1°	3600×2700×75	540	4358	799	10.4	68,000	2437

Scaling and performance are discussed in sec 3 of Kiss et al. 2019; the corresponding plot scripts and data are here. Note that the current 0.1 degree configuration (in table above) requires about half the walltime and CPU hours of the configuration in table 2 of the paper.

Details of run performance are given in .csv tables auto-generated by https://github.com/aekiss/run_summary in /g/data3/hh5/tmp/cosima/access-om2-run-summaries on NCI. Some analysis plots using this data are here.

MOM5 diagnostics list

The list

is here: MOM_diags.txt (best viewed as "raw").

Background

A comprehensive tabulation of MOM5 diagnostics does not exist.

A set of Unix commands to generate such a list is offered on p.413 of S. M. Griffies "Elements of the Modular Ocean Model (MOM) 5 (2012 release with updates)" [Technical Report 7, NOAA/Geophysical Fluid Dynamics Laboratory Ocean Group, February 2012] but this does not deal well with diagnostic registrations that span multiple lines.

The list offered here is more complete, but not without its own problems. Hopefully it will be of some use.

Caveats

This is a work in progress.

Some of these diags may not actually be available because the functions may not be called.

Also many names are programmatically generated so you have to dive into the code to figure out what they are, eg

e:   id_yflux_adv_int_z ( n ) = register_diag_field ( 'ocean_model' , trim ( t_prog ( n ) % name ) // '_yflux_adv_int_z' , grd % tracer_axes_flux_y ( 1 : 2 ) , time % model_time , 'z-integral of cp*rho*dxt*v*temp' , 'Watts' , missing_value = missing_value , range = ( / - 1.e18 , 1.e18 / ) ) 

(we are working on dealing with that case)

How the list was generated

The diag list was produced by Marshall Ward using flint to parse the entire MOM codebase via this script (named parse.py):

   from flint.project import Project

   proj = Project(verbose=True)
   proj.parse('mom5/mom5')

The parse.py script was run (python works fine if 3.x is installed):

   python3 parse.py > out

The parsed output out is then filtered with grep to just list the lines with diag_table field registrations:

   grep -e "^mom5" -e "= *register_diag_field" out > MOM_diags.txt