Skip to content

Commit

Permalink
Merge pull request #760 from MPAS-Dev/develop
Browse files Browse the repository at this point in the history
Merge develop to master
  • Loading branch information
xylar authored Nov 28, 2020
2 parents 00facc6 + 2e1f10a commit cc99370
Show file tree
Hide file tree
Showing 58 changed files with 2,669 additions and 563 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,6 @@ ENV/
.ropeproject

.DS_Store

# Anvil test suite
/anvil_test_suite/
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# MPAS-Analysis
[![Build Status](https://dev.azure.com/MPAS-Dev/MPAS-Analysis%20testing/_apis/build/status/MPAS-Dev.MPAS-Analysis?branchName=refs%2Fpull%2F728%2Fmerge)](https://dev.azure.com/MPAS-Dev/MPAS-Analysis%20testing/_build/latest?definitionId=2&branchName=refs%2Fpull%2F728%2Fmerge)
[![Build Status](https://dev.azure.com/MPAS-Dev/MPAS-Analysis%20testing/_apis/build/status/MPAS-Dev.MPAS-Analysis?branchName=develop)](https://dev.azure.com/MPAS-Dev/MPAS-Analysis%20testing/_build/latest?definitionId=2&branchName=develop)

Analysis for simulations produced with Model for Prediction Across Scales
(MPAS) components and the Energy Exascale Earth System Model (E3SM), which
Expand Down Expand Up @@ -46,7 +46,7 @@ environment with the following packages:
* python >= 3.6
* numpy
* scipy
* matplotlib-base >= 3.0.2
* matplotlib >= 3.0.2
* netCDF4
* xarray >= 0.14.1
* dask
Expand All @@ -62,20 +62,20 @@ environment with the following packages:
* shapely
* cartopy >= 0.18.0
* cartopy\_offlinedata
* geometric\_features >= 0.1.9
* geometric\_features >= 0.1.12
* gsw
* pyremap < 0.1.0
* mpas\_tools >= 0.0.13
* mpas\_tools >= 0.0.15

These can be installed via the conda command:
```
conda config --add channels conda-forge
conda config --set channel_priority strict
conda create -n mpas-analysis python=3.8 numpy scipy "matplotlib-base>=3.0.2" \
conda create -n mpas-analysis python=3.8 numpy scipy "matplotlib>=3.0.2" \
netCDF4 "xarray>=0.14.1" dask bottleneck lxml "nco>=4.8.1" pyproj \
pillow cmocean progressbar2 requests setuptools shapely "cartopy>=0.18.0" \
cartopy_offlinedata "geometric_features>=0.1.9" gsw "pyremap<0.1.0" \
"mpas_tools>=0.0.13"
cartopy_offlinedata "geometric_features>=0.1.12" gsw "pyremap<0.1.0" \
"mpas_tools>=0.0.15"
conda activate mpas-analysis
```

Expand Down
21 changes: 13 additions & 8 deletions azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ jobs:
- bash: |
eval "$(conda shell.bash hook)"
conda create --yes --quiet --name test --use-local python=$PYTHON_VERSION \
mpas-analysis pytest
conda activate build
conda create --yes --quiet --name test -c ${CONDA_PREFIX}/conda-bld/ \
python=$PYTHON_VERSION mpas-analysis pytest
displayName: Create Anaconda test environment
- bash: |
Expand All @@ -58,8 +59,10 @@ jobs:
- bash: |
eval "$(conda shell.bash hook)"
conda create --yes --quiet --name docs --use-local python=$PYTHON_VERSION \
mpas-analysis sphinx mock sphinx_rtd_theme tabulate m2r
conda activate build
conda create --yes --quiet --name docs -c ${CONDA_PREFIX}/conda-bld/ \
python=$PYTHON_VERSION mpas-analysis sphinx mock sphinx_rtd_theme \
tabulate m2r
condition: eq(variables['python.version'], '3.8')
displayName: Create Anaconda docs environment
Expand Down Expand Up @@ -155,8 +158,9 @@ jobs:
- bash: |
eval "$(conda shell.bash hook)"
conda create --yes --quiet --name test --use-local python=$PYTHON_VERSION \
mpas-analysis pytest
conda activate build
conda create --yes --quiet --name test -c ${CONDA_PREFIX}/conda-bld/ \
python=$PYTHON_VERSION mpas-analysis pytest
conda activate test
pip install git+https://github.com/pydata/xarray.git
Expand Down Expand Up @@ -199,8 +203,9 @@ jobs:
- bash: |
eval "$(conda shell.bash hook)"
conda create --yes --quiet --name test --use-local python=$PYTHON_VERSION \
mpas-analysis pytest
conda activate build
conda create --yes --quiet --name test -c ${CONDA_PREFIX}/conda-bld/ \
python=$PYTHON_VERSION mpas-analysis pytest
displayName: Create Anaconda test environment
- bash: |
Expand Down
6 changes: 3 additions & 3 deletions ci/recipe/meta.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{% set name = "MPAS-Analysis" %}
{% set version = "1.2.8" %}
{% set version = "1.2.9" %}

package:
name: {{ name|lower }}
Expand Down Expand Up @@ -40,10 +40,10 @@ requirements:
- shapely
- cartopy >=0.18.0
- cartopy_offlinedata
- geometric_features >=0.1.9
- geometric_features >=0.1.12
- gsw
- pyremap <0.1.0
- mpas_tools >=0.0.13
- mpas_tools >=0.0.15

test:
requires:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## compared against

# mainRunName is a name that identifies the simulation being analyzed.
mainRunName = SO60to10ISC.20200108
mainRunName = GMPAS-IAF.T62_oQU240wLI

# config file for a control run to which this run will be compared. The
# analysis should have already been run to completion once with this config
Expand Down Expand Up @@ -45,17 +45,24 @@ baseDirectory = /lcrc/group/acme/diagnostics
## options related to reading in the results to be analyzed

# directory containing model results
baseDirectory = /lcrc/group/acme/kehoch/acme_scratch/anvil/SO60to10ISC.20200108/run
baseDirectory = /lcrc/group/acme/ac.xylar/acme_scratch/anvil/20201025.GMPAS-IAF.T62_oQU240wLI.anvil

# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oSO60to10wISC
mpasMeshName = oQU240wLI

# subdirectory containing restart files
runSubdirectory = run
# subdirectory for ocean history files
oceanHistorySubdirectory = archive/ocn/hist
# subdirectory for sea ice history files
seaIceHistorySubdirectory = archive/ice/hist

# names of namelist and streams files, either a path relative to baseDirectory
# or an absolute path.
oceanNamelistFileName = mpaso_in
oceanStreamsFileName = streams.ocean
seaIceNamelistFileName = mpassi_in
seaIceStreamsFileName = streams.seaice
oceanNamelistFileName = run/mpaso_in
oceanStreamsFileName = run/streams.ocean
seaIceNamelistFileName = run/mpassi_in
seaIceStreamsFileName = run/streams.seaice

[output]
## options related to writing out plots, intermediate cached data sets, logs,
Expand Down Expand Up @@ -101,9 +108,9 @@ generate = ['all', 'no_BGC', 'no_icebergs', 'no_index', 'no_eke', 'no_min',
## observations and previous runs

# the first year over which to average climatalogies
startYear = 6
startYear = 4
# the last year over which to average climatalogies
endYear = 10
endYear = 8

[timeSeries]
## options related to producing time series plots, often to compare against
Expand Down
5 changes: 2 additions & 3 deletions configs/anvil/job_script.anvil.bash
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,8 @@
cd $SLURM_SUBMIT_DIR
export OMP_NUM_THREADS=1

source /home/xylar/miniconda3/etc/profile.d/conda.sh
conda activate mpas-analysis
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified.sh
export HDF5_USE_FILE_LOCKING=FALSE

srun -N 1 -n 1 python -m mpas_analysis configs/polarRegions.conf config.SO60to10ISC.20200108
srun -N 1 -n 1 python -m mpas_analysis configs/polarRegions.conf 20201025.GMPAS-IAF.T62_oQU240wLI.anvil.cg

150 changes: 150 additions & 0 deletions configs/anvil/test_suite/QU480.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
[runs]
## options related to the run to be analyzed and control runs to be
## compared against

# mainRunName is a name that identifies the simulation being analyzed.
mainRunName = QU480

# config file for a control run to which this run will be compared. The
# analysis should have already been run to completion once with this config
# file, so that the relevant MPAS climatologies already exist and have been
# remapped to the comparison grid. Leave this option commented out if no
# control run is desired.
# controlRunConfigFile = /path/to/config/file

# config file for a main run on which the analysis was already run to
# completion. The relevant MPAS climatologies already exist and have been
# remapped to the comparison grid and time series have been extracted.
# Leave this option commented out if the analysis for the main run should be
# performed.
# mainRunConfigFile = /path/to/config/file

[execute]
## options related to executing parallel tasks

# the number of parallel tasks (1 means tasks run in serial, the default)
parallelTaskCount = 12

# the parallelism mode in ncclimo ("serial" or "bck")
# Set this to "bck" (background parallelism) if running on a machine that can
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = bck

[diagnostics]
## config options related to observations, mapping files and region files used
## by MPAS-Analysis in diagnostics computations.

# The base path to the diagnostics directory. Typically, this will be a shared
# directory on each E3SM supported machine (see the example config files for
# its location). For other machines, this would be the directory pointed to
# when running "download_analysis_data.py" to get the public observations,
# mapping files and region files.
baseDirectory = /lcrc/group/acme/diagnostics

[input]
## options related to reading in the results to be analyzed

# directory containing model results
baseDirectory = /lcrc/group/acme/ac.xylar/acme_scratch/anvil/20200305.A_WCYCL1850.ne4_oQU480.anvil

# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oQU480

# subdirectory containing restart files
runSubdirectory = run
# subdirectory for ocean history files
oceanHistorySubdirectory = archive/ocn/hist
# subdirectory for sea ice history files
seaIceHistorySubdirectory = archive/ice/hist

# names of namelist and streams files, either a path relative to baseDirectory
# or an absolute path.
oceanNamelistFileName = run/mpaso_in
oceanStreamsFileName = run/streams.ocean
seaIceNamelistFileName = run/mpassi_in
seaIceStreamsFileName = run/streams.seaice

[output]
## options related to writing out plots, intermediate cached data sets, logs,
## etc.

# directory where analysis should be written
baseDirectory = /lcrc/group/acme/ac.xylar/analysis_testing/baseline

# Anvil doesn't have direct access to a web portal, so output will need
# to be copied elsewhere (e.g. NERSC web portal)
htmlSubdirectory = /lcrc/group/acme/public_html/diagnostic_output/ac.xylar/analysis_testing/baseline

# a list of analyses to generate. Valid names can be seen by running:
# mpas_analysis --list
# This command also lists tags for each analysis.
# Shortcuts exist to generate (or not generate) several types of analysis.
# These include:
# 'all' -- all analyses will be run
# 'all_<tag>' -- all analysis with a particular tag will be run
# 'all_<component>' -- all analyses from a given component (either 'ocean'
# or 'seaIce') will be run
# 'only_<component>', 'only_<tag>' -- all analysis from this component or
# with this tag will be run, and all
# analysis for other components or
# without the tag will be skipped
# 'no_<task_name>' -- skip the given task
# 'no_<component>', 'no_<tag>' -- in analogy to 'all_*', skip all analysis
# tasks from the given compoonent or with
# the given tag. Do
# mpas_analysis --list
# to list all task names and their tags
# an equivalent syntax can be used on the command line to override this
# option:
# mpas_analysis config.analysis --generate \
# all,no_ocean,all_timeSeries
# All tasks with tag "landIceCavities" are disabled because this run did not
# include land-ice cavities.
generate = ['all', 'no_BGC', 'no_icebergs', 'no_index', 'no_eke',
'no_landIceCavities']

[climatology]
## options related to producing climatologies, typically to compare against
## observations and previous runs

# the first year over which to average climatalogies
startYear = 3
# the last year over which to average climatalogies
endYear = 5

[timeSeries]
## options related to producing time series plots, often to compare against
## observations and previous runs

# start and end years for timeseries analysis. Use endYear = end to indicate
# that the full range of the data should be used. If errorOnMissing = False,
# the start and end year will be clipped to the valid range. Otherwise, out
# of bounds values will lead to an error. In a "control" config file used in
# a "main vs. control" analysis run, the range of years must be valid and
# cannot include "end" because the original data may not be available.
startYear = 1
endYear = end

[index]
## options related to producing nino index.

# start and end years for El Nino 3.4 analysis. Use endYear = end to indicate
# that the full range of the data should be used. If errorOnMissing = False,
# the start and end year will be clipped to the valid range. Otherwise, out
# of bounds values will lead to an error. In a "control" config file used in
# a "main vs. control" analysis run, the range of years must be valid and
# cannot include "end" because the original data may not be available.
startYear = 1
endYear = end

[streamfunctionMOC]
## options related to plotting the streamfunction of the meridional overturning
## circulation (MOC)

# Use postprocessing script to compute the MOC? You want this to be True
# for low-resolution simulations that use GM to parameterize eddies, because
# the online MOC analysis member currently does not include the bolus velocity
# in its calculation, whereas the postprocessing script does.
# NOTE: this is a temporary option that will be removed once the online
# MOC takes into account the bolus velocity when GM is on.
usePostprocessingScript = True
9 changes: 9 additions & 0 deletions configs/anvil/test_suite/clean_suilte.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash

set -e

branch=$(git symbolic-ref --short HEAD)

rm -rf anvil_test_suite
rm -rf /lcrc/group/acme/ac.xylar/analysis_testing/${branch}
rm -rf /lcrc/group/acme/public_html/diagnostic_output/ac.xylar/analysis_testing/${branch}
Loading

0 comments on commit cc99370

Please sign in to comment.