Skip to content

Latest commit

 

History

History
522 lines (361 loc) · 15.9 KB

course.rst

File metadata and controls

522 lines (361 loc) · 15.9 KB
Raw

Running ACCESS-OM2 Models at NCI

subtitle:Using payu to run ACCESS-OM2 models
Author: Aidan Heerdegen
description:A training course to introduce running ACCESS-OM2 models using payu
Date: 18 May 2021

Outline

  • ACCESS-OM2 Model
  • Configurations
  • Running the model
  • Other useful stuff

ACCESS-OM2

Model suite includes 1, 0.25 and 0.1 degree global Ocean/Sea Ice models forced with atmospheric data and almost identical model parameters.

Single access-om2 repository with all code and configs

https://github.com/COSIMA/access-om2

Well documented in wiki

https://github.com/COSIMA/access-om2/wiki

Components

The model consists of separate component models that run independently and share information via a coupler

Ocean MOM5
Ice CICE5
Atmosphere libaccessom2 + yatm
Coupler OASIS3-MCT

Code

All models are open source

MOM5 https://github.com/mom-ocean/MOM5
CICE5 https://github.com/COSIMA/cice5
libaccessom2 https://github.com/COSIMA/libaccessom2
OASIS3-MCT https://github.com/COSIMA/oasis3-mct

Forcing Data

  • Atmosphere is not free-running model
  • Atmospheric forcing from data source. Read by libaccessom2+yatm, passed to ice model
  • Atmosphere uses JRA55 reanalysis derivative product JRA55-do v1.4

http://jra.kishou.go.jp/JRA-55/index_en.html https://www.sciencedirect.com/science/article/pii/S146350031830235X

  • IAF (Interannual Forcing) : JRA55-do (1955-present)
  • RYF (Repeat Year Forcing) : RYF9091 (RYF8485, RYF0304 also available)

Configurations

All model configurations are global, and there are three supported resolutions

At each resolution interannual and repeat year forcing is supported

COSIMA Model name: ACCESS-OM2

  • Nominal 1 degree global resolution
  • JRA55 RYF and IAF

https://github.com/COSIMA/1deg_jra55_iaf https://github.com/COSIMA/1deg_jra55_ryf

COSIMA Model name: ACCESS-OM2-025

  • Nominal 0.25 degree global resolution
  • JRA55 RYF and IAF configurations

https://github.com/COSIMA/025deg_jra55_ryf https://github.com/COSIMA/025deg_jra55_iaf

.. notes::
   Don't suggest anyone runs this without contacting COSIMA
     as runs are expensive

COSIMA Model name: ACCESS-OM2-01

  • Nominal 0.1 degree global resolution
  • JRA55 RYF and IAF configurations

https://github.com/COSIMA/01deg_jra55_iaf https://github.com/COSIMA/01deg_jra55_ryf

Running the model

.. notes::
   Can run in a branch to keep config clean
   Can fork on GitHub and push config changes back to fork

Follow the Quick Start instructions in the ACCESS-OM2 Wiki on github

https://github.com/COSIMA/access-om2/wiki/Getting-started#quick-start

As an example, using the 1 deg JRA55 IAF configuration:

1deg_jra55_iaf/
|-- accessom2.nml
|-- atmosphere
|   |-- atm.nml
|   `-- forcing.json
|-- config.yaml
|-- doc
|   `-- Fabio2018_Namelist_meeting_final.pdf
|-- ice
|   |-- cice_in.nml
|   |-- input_ice_gfdl.nml
|   |-- input_ice_monin.nml
|   `-- input_ice.nml
|-- manifests
|   |-- exe.yaml
|   |-- input.yaml
|   `-- restart.yaml
|-- metadata.yaml
|-- namcouple
|-- nmltab.py
|-- ocean
|   |-- data_table
|   |-- diag_table
|   |-- diag_table_source.yaml
|   |-- field_table
|   |-- input.nml
|   `-- make_diag_table.py
|-- README.md
|-- resub.sh
|-- run_summary.py
|-- sync_data.sh
`-- tidy_restarts.py

The libaccessom2 namelist accessom2.nml controls logging, timesetp, forcing dates and model run length (restart_period)

To run the model as a test change model run time from 5 years to 1 month: in accessom2.nml

Run the model

  • payu prints the the command it submits and other information
  • Some errors visible by inspection: incorrect payu path, incorrect storage flags, inefficient processor count

The PBS options for normal queue

The model options

Collation options include collating restarts, and using multiple CPUs to speed up collation

Miscellaneous options

Restart from another experiment

  • payu will use any existing restart in the archive directory
  • restart option in config.yaml is best, but doesn't currently work for ACCESS-OM2
  • Copy or link previous restart and output directories into your archive. See the ACCESS-OM2 wiki for details

Has my model crashed?

Signs your model has crashed:

  • work/ directory still present in control directory
  • access-om2.out and access-om2.err still present in control directory
  • Non-zero Exit Status in PBS log file
======================================================================================
                Resource Usage on 2021-05-25 17:02:12:
Job Id:             22990186.gadi-pbs
Project:            w97
Exit Status:        1
Service Units:      6.88
NCPUs Requested:    288                    NCPUs Used: 288
                                        CPU Time Used: 00:21:24
Memory Requested:   1000.0GB              Memory Used: 121.07GB
Walltime requested: 03:00:00            Walltime Used: 00:00:43
JobFS requested:    600.0MB                JobFS used: 0B
======================================================================================

Diagnosing a crash

.. notes::
   The code was compiled to include human readable source code names which makes it easier to debug

   This is an MPI program running multiple copies of
   the models, and each will be signalled to "die" when there is a fatal error
  • Check access-om2.err and find the first error message that looks like this
  • Backtrace (or stacktrace) shows location in the code where error occurred
  • A stack trace for every cpu (rank)

Another example of an error

  • If the cause of the crash isn't obvious might have to dig into log files within the model subdirectories in work/

Other Useful Stuff

Diagnostics

  • Only a fraction of the possible diagnostic (and tracer) fields are output
  • MOM diagnostics determined by the diag_table which is generated programmatically
  • CICE diagnostics are definted in cice_in.nml

Perturbations

Add offsets or scale atmospheric forcing fields. Either single value, spatially and/or temporally varying e.g.

See the wiki for details

https://github.com/COSIMA/access-om2/wiki/Tutorials#Scaling-the-forcing-fields

Available data

  • Some data is published and available via THREDDS from NCI

http://dx.doi.org/10.4225/41/5a2dc8543105a

  • Always preferable (faster) to access directly on disk
  • Need to join groups: hh5, ik11, cj50 and qv56 at https://my.nci.org.au

Analysis

COSIMA provides the COSIMA Cookbook, a database to find and load COSIMA datasets

COSIMA Recipes repository contains Tutorials and Documented Examples

Cookbook includes interactive Data Explorer tool to find and load COSIMA datasets at NCI

Compiling Executables

  • The models are compiled and then published to a shared area in ik11
  • Build scripts Andrew Kiss has developed link the codebase to the executable name via git commit hashes
  • Best to use published executables with known provenance
  • If compiling executables keep access-om2 source tree separate from your control directories