ufs-community · MichaelLueken · Oct 31, 2023 · Oct 26, 2023 · Oct 26, 2023 · Oct 26, 2023
@@ -15,7 +15,7 @@ https://epic.noaa.gov/wp-content/uploads/2022/12/Debugging-Guide.pdf
 
 The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
 
-UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
+UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
 
 [![Python unittests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml)
 [![Python functional tests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml)

@@ -417,6 +417,19 @@ else
 fi
 module list
 
+# Apply patch for sorc/CMakeLists.txt for MacOS arm64/aarch64
+if [[ "${PLATFORM}" == "macos" ]]; then
+  ARCH=$(uname -m)
+  if [[ "${ARCH}" == arm64 ]] || [[ "${ARCH}" == aarch64 ]]; then
+    patch1="sorc/patch_macos_arm64_sorc_cmakelists.txt"
+    if patch -p1 -R --dry-run --silent -d ./sorc -N < ${patch1} 1> /dev/null; then
+      echo "Patch ${patch1} was already applied";
+    else
+      patch -p1 -d ./sorc -N < ${patch1}
+    fi
+  fi
+fi
+
 mkdir -p ${BUILD_DIR}
 cd ${BUILD_DIR}
 

@@ -31,7 +31,7 @@ Since the last release, developers have added a variety of features:
 
 The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
 
-UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
+UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
 
 User's Guide Organization 
 ============================

@@ -15,7 +15,7 @@ The Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application
 
 To build the SRW App, users will complete the following steps:
 
-   #. :ref:`Install prerequisites <HPCstackInfo>`
+   #. :ref:`Install prerequisites <StackInfo>`
    #. :ref:`Clone the SRW App from GitHub <DownloadSRWApp>`
    #. :ref:`Check out the external repositories <CheckoutExternals>`
    #. :ref:`Set up the build environment and build the executables <BuildExecutables>`
@@ -358,9 +358,8 @@ Additional Details for Building on MacOS or Generic Linux
 
 The SRW App can be built on MacOS and generic Linux machines after the prerequisite software has been installed on these systems (via :term:`HPC-Stack` or :term:`spack-stack`). The installation for MacOS is architecture-independent and has been tested using both x86_64 and M1 chips (running natively). The following configurations for MacOS have been tested:
 
-   #. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), Monterey Sur 12.1, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
-   #. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, Big Sur 11.6.4, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
-   #. MacBook Pro 2015, 2.8 GHz Quad-Core Intel Core i7 (x86_64), Catalina OS X 10.15.7, GNU compiler suite v.11.2.0_3 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
+   #. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), OS Monterey 12.6.1, 32 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
+   #. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, OS Ventura 13.0.1, 16 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
 
 Several Linux builds have been tested on systems with x86_64 architectures.
 

@@ -80,19 +80,23 @@ On most Level 1 systems, a container named ``ubuntu20.04-intel-srwapp-release-pu
 
    * - Machine
      - File Location
-   * - Cheyenne/Derecho
+   * - Cheyenne/Derecho `*`_
      - /glade/scratch/epicufsrt/containers
-   * - Gaea
+   * - Gaea `*`_
      - /lustre/f2/dev/role.epic/containers
    * - Hera
      - /scratch1/NCEPDEV/nems/role.epic/containers
    * - Jet
      - /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers
    * - NOAA Cloud
      - /contrib/EPIC/containers
-   * - Orion/Hercules
+   * - Orion/Hercules `*`_
      - /work/noaa/epic/role-epic/contrib/containers
 
+.. _`*` : 
+
+   \* On these systems, container testing shows inconsistent results. 
+
 .. note::
    * On Gaea, Singularity/Apptainer is only available on the C5 partition, and therefore container use is only supported on Gaea C5. 
    * The NOAA Cloud containers are accessible only to those with EPIC resources. 

@@ -192,13 +192,79 @@ MacOS requires the installation of a few additional packages and, possibly, an u
 Creating the |wflow_env| Environment on Linux and Mac OS
 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
 
-On generic Mac and Linux systems, users need to create a conda |wflow_env| environment. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) The following is a brief recipe for creating a virtual conda environment on non-Level 1 platforms. It uses the aarch64 (64-bit ARM) Miniforge for Linux and installs into $HOME/conda. Adjust as necessary for your target system.
+On generic Mac and Linux systems, users need to create a conda |wflow_env| environment that contains python package required for running the workflow. Other conda environments may need to be activated for running graphics generation tasks, |graphics_env|, or when testing the AQM/CMAQ, |cmaq_env|. Python packages in these other environments  that may conflict with those in |wflow_env|. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) 
+These conda environments could be added to the existing python or conda modules. This is needed for non-Level 1 platforms. 
+There are several options available for building virtual conda environments on non-Level 1 platforms. The examples in this section use the aarch64 (64-bit ARM) Miniforge for Linux and install into $HOME/conda. Users should adjust as needed for their target system. 
+
+Options include:
+
+1) configuration files in *.yaml format that list core packages required for each of the environments; 2) a more general recipe for installation of the Miniforge3 package that then installs packages for the |wflow-env| as listed in repository workflow-tools (https://github.com/ufs-community/workflow-tools); and 
+3) installing a miniconda3 module (Lmod) and creating three environments |wflow_env|, |graphics_env|, |cmaq_env|.
+
+1) workfow_tools.yaml for |wflow_env|
+
+.. code-block:: console
+
+   name: workflow_tools
+   channels:
+     - conda-forge
+     - defaults
+   dependencies:
+     - python=3.9*
+     - boto3=1.22*
+     - black
+     - f90nml=1.4*
+     - jinja2=3.0*
+     - numpy=1.21*
+     - pylint
+     - pytest
+     - pyyaml=6.0*
+     - tox  
+
+
+regional_workflow.yaml for |graphics_env|
 
 .. code-block:: console
 
-   wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
-   bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
-   rm Miniforge3-Linux-aarch64.sh
+   name: regional_workflow
+   channels:
+     - conda-forge
+     - defaults
+   dependencies:
+     - python=3.9.*
+     - f90nml
+     - jinja2
+     - pyyaml
+     - scipy
+     - matplotlib=3.5.2*
+     - pygrib
+     - cartopy
+
+
+.. code-block:: console
+
+   name: regional_workflow_cmaq
+   channels:
+     - conda-forge
+     - defaults
+   dependencies:
+     - python=3.9.12
+     - f90nml=1.4*
+     - jinja2=3.0*
+     - pyyaml=6.0*
+     - scipy
+     - matplotlib
+     - pygrib
+     - cartopy
+     - netcdf4
+     - xarray
+
+2) Installing the Miniforge, as example for the aarch64 (64-bit ARM) Miniforge for Linux that installs into $HOME/conda. Adjust as necessary for your target system. 
+
+
+      wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
+      bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
+      rm Miniforge3-Linux-aarch64.sh
    source ~/conda/etc/profile.d/conda.sh
    conda activate
    conda install -y conda-build conda-verify
@@ -216,6 +282,9 @@ In future shells, you can activate and use this environment with:
 
 See the `workflow-tools repository <https://github.com/ufs-community/workflow-tools>`__ for additional documentation. 
 
+3)  Building miniconda3 and creating Lmod modulefile loadable among other modules during the workflow. The module could be added to your <wflow>_platform.lua modulefile, and the environments activated or deactivated as needed for a particular workflow task. A repository with full installation instructions, a modulefile template, and environment configuration files could be accessed here: https://github.com/NOAA-EPIC/miniconda3. Full instructions could be viewd  in a README.md file ( https://github.com/NOAA-EPIC/miniconda3/edit/master/README.md )
+
+
 Modify a ``wflow_<platform>`` File
 ``````````````````````````````````````
 
@@ -632,7 +701,7 @@ This can be helpful when conducting multiple experiments with different types of
 Plotting Configuration (optional)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
+An optional Python plotting task (plot_allvars) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
 output over the :term:`CONUS`. It generates graphics plots for a number of variables, including:
 
    * 2-m temperature

@@ -164,8 +164,8 @@ If users wish to modify the fields or levels that are output from the UPP, they
    This process requires advanced knowledge of which fields can be output for the UFS Weather Model.
 
 UPP Product Output Tables for the UFS SRW LAM Grid:
-   * :doc:`3D Native Hybrid Level Fields <tables/SRW_NATLEV_table>`
-   * :doc:`3D Pressure Level Fields <tables/SRW_PRSLEV_table>`
+   * :ref:`3D Native Hybrid Level Fields <SRW_NATLEV_table>`
+   * :ref:`3D Pressure Level Fields <SRW_PRSLEV_table>`
 
 Use the instructions in the `UPP User's Guide <https://upp.readthedocs.io/en/upp-srw-v2.2.0-docs/InputsOutputs.html#control-file>`__ to make modifications to the ``fv3lam.xml`` file and to remake the flat text file, called ``postxconfig-NT-fv3lam.txt`` (default), that the UPP reads.
 

@@ -90,6 +90,8 @@
 rst_prolog = """
 .. |wflow_env| replace:: ``workflow_tools``
 .. |activate| replace:: ``conda activate workflow_tools``
+.. |graphic_env| replace:: ``regional_workflow``
+.. |aqm_env| replace:: ``regional_workflow_aqm``
 .. |prompt| replace:: ``(workflow_tools)``
 .. |latestr| replace:: v2.2.0
 .. |branch| replace:: ``release/public-v2.2.0``

@@ -1,5 +1,7 @@
 :orphan:
 
+.. _SRW_NATLEV_table:
+
 ************************************************************
 Fields Requested in the UPP Parameter Table for SRW NATLEV
 ************************************************************

@@ -1,5 +1,7 @@
 :orphan:
 
+.. _SRW_PRSLEV_table:
+
 **********************************************************
 Fields Requested in the UPP Parameter Table for SRW PRSLEV
 **********************************************************

@@ -18,22 +18,23 @@ if ( "$L_MACHINE" != wcoss2 ) then
 endif
 
 if ( "$L_MACHINE" == macos ) then
-   arch=$(uname -m)
-   [[ "$arch" = arm64 ]] && export ENV="/opt/homebrew/opt/lmod/init/csh"
-   [[ "$arch" = x86_64 ]] && export ENV="/usr/local/opt/lmod/init/csh"
-   source $ENV
+   if ( -d /opt/homebrew/opt/lmod ) then
+       source /opt/homebrew/opt/lmod/init/csh
+   else if ( -d /usr/local/opt/lmod ) then
+       source /usr/local/opt/lmod/init/csh
+   else
+       echo "Path for Lmod is unknown, verify whether Lmod is installed"
+   endif
 
    module purge
 
 else if ( "$L_MACHINE" == linux ) then
-   setenv ENV "/usr/share/lmod/lmod/init/csh"
-   source $ENV
+   source /usr/share/lmod/lmod/init/csh
 
    module purge
 
 else if ( "$L_MACHINE" == singularity ) then
-   set ENV="/usr/share/lmod/lmod/init/csh"
-   source $ENV
+   source /usr/share/lmod/lmod/init/csh
 
    module purge
 

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
 
 if [ $# = 0 ]; then
    L_MACHINE=${MACHINE}
@@ -26,10 +26,12 @@ fi
 
 if [ "$L_MACHINE" = macos ]; then
    arch=$(uname -m)
-   [[ "$arch" = arm64 ]] && export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
-   [[ "$arch" = x86_64 ]] && export BASH_ENV="/usr/local/opt/lmod/init/bash"
+   if [ "$arch" = arm64 ] || [ "$arch" = aarch64 ]; then
+      export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
+   else
+      export BASH_ENV="/usr/local/opt/lmod/init/bash"
+   fi
    source $BASH_ENV
-
    module purge
 
 elif [ "$L_MACHINE" = linux ]; then

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------