Merge pull request #843 from MPAS-Dev/develop

Merge develop to master for 1.5.0 release

README.md

@@ -64,7 +64,7 @@ package in `dev-spec.txt`.  Then, you can install both packages in the same
 development environment, e.g.:
 ``` bash
 conda create -y -n mpas_dev --file tools/MPAS-Tools/conda_package/dev-spec.txt \
-    -- analysis/MPAS-Analysis/dev-spec.txt
+    --file analysis/MPAS-Analysis/dev-spec.txt
 conda activate mpas_dev
 cd tools/MPAS-Tools/conda_package
 python -m pip install -e .
@@ -109,7 +109,7 @@ mpas_analysis --list
 ```
 
 This lists all tasks and their tags.  These can be used in the `generate`
-command-line option or config option.  See `mpas_analysis/config.default`
+command-line option or config option.  See `mpas_analysis/default.cfg`
 for more details.
 
 ## Running the analysis
@@ -119,17 +119,17 @@ for more details.
      git repo) or download one from the
      [example configs directory](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/configs).
   2. Either modify config options in your new file or copy and modify config
-     options from `mpas_analysis/config.default` (in a git repo) or directly
+     options from `mpas_analysis/default.cfg` (in a git repo) or directly
      from GitHub:
-     [config.default](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/mpas_analysis/config.default).
+     [default.cfg](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/mpas_analysis/default.cfg).
   3. If you installed the `mpas-analysis` package, run:
      `mpas_analysis config.myrun`.  This will read the configuration
-     first from `mpas_analysis/config.default` and then replace that
+     first from `mpas_analysis/default.cfg` and then replace that
      configuraiton with any changes from from `config.myrun`
   4. If you want to run a subset of the analysis, you can either set the
      `generate` option under `[output]` in your config file or use the
      `--generate` flag on the command line.  See the comments in
-     `mpas_analysis/config.default` for more details on this option.
+     `mpas_analysis/default.cfg` for more details on this option.
 
   **Requirements for custom config files:**
   * At minimum you should set `baseDirectory` under `[output]` to the folder
@@ -139,10 +139,10 @@ for more details.
     updated correctly.
   * Any options you copy into the config file **must** include the
     appropriate section header (e.g. '[run]' or '[output]')
-  * You do not need to copy all options from `mpas_analysis/config.default`.
+  * You do not need to copy all options from `mpas_analysis/default.cfg`.
     This file will automatically be used for any options you do not include
     in your custom config file.
-  * You should **not** modify `mpas_analysis/config.default` directly.
+  * You should **not** modify `mpas_analysis/default.cfg` directly.
 
 ## List of MPAS output files that are needed by MPAS-Analysis:
 
@@ -240,7 +240,7 @@ already makes or creating new ones:
 
 1. customize the config file. Some features, such as colormaps and colorbar
    limits for color shaded plot or depth ranges for ocean region time series,
-   can be customized: look at `mpas_analysis/config.default` for available
+   can be customized: look at `mpas_analysis/default.cfg` for available
    customization for each analysis task.
 2. read in the analysis data computed by MPAS-Analysis into custom scripts. When
    running MPAS-Analysis with the purpose of generating both climatologies
@@ -268,7 +268,7 @@ within the `mpas_analysis/shared` directory.
    described in the template.  Take a look at
    `mpas_analysis/shared/analysis_task.py` for additional guidance.
 2. note, no changes need to be made to `mpas_analysis/shared/analysis_task.py`
-3. modify `mpas_analysis/config.default` (and possibly any machine-specific
+3. modify `mpas_analysis/default.cfg` (and possibly any machine-specific
    config files in `configs/<machine>`)
 4. import new analysis task in `mpas_analysis/<component>/__init__.py`
 5. add new analysis task to `mpas_analysis/__main__.py` under

ci/recipe/meta.yaml

@@ -1,5 +1,5 @@
 {% set name = "MPAS-Analysis" %}
-{% set version = "1.4.0" %}
+{% set version = "1.5.0" %}
 
 package:
   name: {{ name|lower }}
@@ -18,10 +18,10 @@ build:
 
 requirements:
   host:
-    - python >=3.7
+    - python >=3.7,<3.10
     - pip
   run:
-    - python >=3.7
+    - python >=3.7,<3.10
     - bottleneck
     - cartopy >=0.18.0
     - cartopy_offlinedata
@@ -30,6 +30,7 @@ requirements:
     - geometric_features >=0.5.0
     - gsw
     - lxml
+    - mache >=1.1.2
     - matplotlib-base >=3.0.2
     - mpas_tools >=0.5.1
     - nco >=4.8.1

configs/README.md

@@ -12,11 +12,10 @@ job script or the default job script.
 ## Analysis focused on polar regions
 
 For analysis such as E3SM cryosphere simulations that is focused on polar
-regions, include polarRegions.conf *before* your custom config file on the
-command line.
+regions, include the `--polar_regions` flag on the command line.
 
 Example:
 ```
-mpas_analysis configs/polarRegions.conf my_favorite.conf --generate=climatologyMapSSH --purge
+mpas_analysis --polar_regions --generate=climatologyMapSSH --purge my_favorite.cfg 
 ```
 

configs/compy/test_suite/ctrl.cfg

@@ -19,28 +19,6 @@ mainRunName = ctrl
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/compy/test_suite/job_script.bash

@@ -14,14 +14,15 @@ export OMP_NUM_THREADS=1
 source ${HOME}/miniconda3/etc/profile.d/conda.sh
 conda activate test_env
 export HDF5_USE_FILE_LOCKING=FALSE
+export E3SMU_MACHINE=compy
 
 echo env: test_env
-echo configs: ../../configs/polarRegions.conf ../main.cfg
+echo configs: --polar_regions ../main.cfg
 
 srun -N 1 -n 1 mpas_analysis --list
 srun -N 1 -n 1 mpas_analysis --plot_colormaps
-srun -N 1 -n 1 mpas_analysis --setup_only ../../configs/polarRegions.conf ../main.cfg
-srun -N 1 -n 1 mpas_analysis --purge ../../configs/polarRegions.conf ../main.cfg --verbose
-srun -N 1 -n 1 mpas_analysis --html_only ../../configs/polarRegions.conf ../main.cfg
+srun -N 1 -n 1 mpas_analysis --setup_only --polar_regions ../main.cfg
+srun -N 1 -n 1 mpas_analysis --purge --polar_regions ../main.cfg --verbose
+srun -N 1 -n 1 mpas_analysis --html_only --polar_regions ../main.cfg
 
 chmod -R ugo+rX /compyfs/www/asay932/analysis_testing/

configs/compy/test_suite/job_script_no_polar_regions.bash

@@ -14,6 +14,7 @@ export OMP_NUM_THREADS=1
 source ${HOME}/miniconda3/etc/profile.d/conda.sh
 conda activate test_env
 export HDF5_USE_FILE_LOCKING=FALSE
+export E3SMU_MACHINE=compy
 
 echo env: test_env
 echo configs: ../no_polar_regions.cfg

configs/compy/test_suite/main.cfg

@@ -19,28 +19,6 @@ mainRunName = main
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/compy/test_suite/main_vs_ctrl.cfg

@@ -17,29 +17,7 @@ controlRunConfigFile = ../ctrl.cfg
 # remapped to the comparison grid and time series have been extracted.
 # Leave this option commented out if the analysis for the main run should be
 # performed.
-mainRunConfigFile = ../main_py3.8.cfg
-
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
+mainRunConfigFile = ../main_py3.9.cfg
 
 [input]
 ## options related to reading in the results to be analyzed

configs/compy/test_suite/mesh_rename.cfg

@@ -19,28 +19,6 @@ mainRunName = main
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/compy/test_suite/no_ncclimo.cfg

@@ -19,28 +19,6 @@ mainRunName = main
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/compy/test_suite/test_suite.bash

@@ -14,16 +14,16 @@ conda activate base
 
 branch=$(git symbolic-ref --short HEAD)
 
-conda update -y conda conda-build
+conda update -y conda conda-build mamba boa
 rm -rf ${HOME}/miniconda3/conda-bld
-conda build ci/recipe
+conda mambabuild ci/recipe
 
 # create the test conda envs
 for py in ${main_py} ${alt_py}
 do
     env=test_mpas_analysis_py${py}
     conda remove -y --all -n ${env}
-    conda create -y -n ${env} --use-local python=${py} mpas-analysis sphinx \
+    mamba create -y -n ${env} --use-local python=${py} mpas-analysis sphinx \
         mock sphinx_rtd_theme "tabulate>=0.8.2" m2r pytest
     conda activate ${env}
     pytest
@@ -33,7 +33,7 @@ done
 # create another env for testing xarray master branch
 py=${main_py}
 env=test_mpas_analysis_xarray_master
-conda create --yes --quiet --name ${env} --use-local python=${py} \
+mamba create --yes --quiet --name ${env} --use-local python=${py} \
     mpas-analysis pytest
 conda activate ${env}
 pip install git+https://github.com/pydata/xarray.git

configs/compy/test_suite/wc_defaults.cfg

@@ -19,28 +19,6 @@ mainRunName = wc_defaults
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 6
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /compyfs/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/cori/test_suite/ctrl.cfg

@@ -19,28 +19,6 @@ mainRunName = ctrl
 # performed.
 # mainRunConfigFile = /path/to/config/file
 
-[execute]
-## options related to executing parallel tasks
-
-# the number of parallel tasks (1 means tasks run in serial, the default)
-parallelTaskCount = 12
-
-# the parallelism mode in ncclimo ("serial" or "bck")
-# Set this to "bck" (background parallelism) if running on a machine that can
-# handle 12 simultaneous processes, one for each monthly climatology.
-ncclimoParallelMode = bck
-
-[diagnostics]
-## config options related to observations, mapping files and region files used
-## by MPAS-Analysis in diagnostics computations.
-
-# The base path to the diagnostics directory.  Typically, this will be a shared
-# directory on each E3SM supported machine (see the example config files for
-# its location).  For other machines, this would be the directory pointed to
-# when running "download_analysis_data.py" to get the public observations,
-# mapping files and region files.
-baseDirectory = /global/cfs/cdirs/e3sm/diagnostics
-
 [input]
 ## options related to reading in the results to be analyzed
 

configs/cori/test_suite/job_script.bash

@@ -17,14 +17,15 @@ export OMP_NUM_THREADS=1
 source ${HOME}/miniconda3/etc/profile.d/conda.sh
 conda activate test_env
 export HDF5_USE_FILE_LOCKING=FALSE
+export E3SMU_MACHINE=cori-haswell
 
 echo env: test_env
-echo configs: ../../configs/polarRegions.conf ../main.cfg
+echo configs: --polar_regions ../main.cfg
 
 srun -N 1 -n 1 mpas_analysis --list
 srun -N 1 -n 1 mpas_analysis --plot_colormaps
-srun -N 1 -n 1 mpas_analysis --setup_only ../../configs/polarRegions.conf ../main.cfg
-srun -N 1 -n 1 mpas_analysis --purge ../../configs/polarRegions.conf ../main.cfg --verbose
-srun -N 1 -n 1 mpas_analysis --html_only ../../configs/polarRegions.conf ../main.cfg
+srun -N 1 -n 1 mpas_analysis --setup_only --polar_regions ../main.cfg
+srun -N 1 -n 1 mpas_analysis --purge --polar_regions ../main.cfg --verbose
+srun -N 1 -n 1 mpas_analysis --html_only --polar_regions ../main.cfg
 
 chmod -R ugo+rX /global/cfs/cdirs/e3sm/www/xylar/analysis_testing/