Replace manage_externals with git-fleximod (#276)

**Originator(s):** nusbaume
**Date:**  July 7th, 2024
**One-line Summary:**  Replace manage_externals with git-fleximod

**Purpose of changes:**  

This PR removes the `manage_externals` tool and replaces it with the new
`git-fleximod` system, which is what will be used in CAM7 and CESM3.
This PR also introduces an SE-CSLAM dycore build test to the CAM-SIMA
regression test list.

**Issues closed (include Github closing keyword and issue number):**

Closes #266 
Closes #275 

**Describe any changes made to build system:**  

Externals are now brought in via `bin/git-fleximod update` instead of
`manage_externals/checkout_externals`.

**Describe any changes made to the namelist:**  None

**List any changes to the defaults for the boundary datasets:** None

**Describe any substantial timing or memory changes:** None

**Code reviewed by:** cacraigucar, peverwhee

**List all files eliminated:**

D       Externals.cfg
D       Externals_CAM.cfg
D       manage_externals/*
  - Remove manage_externals tool and associated files

**List all files added and what they do:**

A       .gitmodules
A       bin/git-fleximod
A       .lib/git-fleximod/*
A       external git submodules
  - Add git-fleximod tool and associated files and submodules.

A
cime_config/testdefs/testmods_dirs/cam/outfrq_kessler_se_cslam_derecho_nooutput/shell_commands
A
cime_config/testdefs/testmods_dirs/cam/outfrq_kessler_se_cslam_derecho_nooutput/user_nl_cam
  - Add new SE-CSLAM dycore build test to regression test list.   
 
**List all existing files that have been modified, and describe the
changes:**

M       .github/workflows/python_unit_tests.yml
  - Use git-fleximod instead of manage_externals in testing workflow.

M       .github/scripts/pr_mod_file_tests.py
  - Ignore git-fleximod files when performing static analysis.

M       .gitignore
  - Update .gitignore to properly work with git submodules

M       cime_config/cam_autogen.py
  - Fix failing doctest

M       cime_config/testdefs/testlist_cam.xml
  - Add SE-CSLAM build test to regression test list.


### Test results: 

**derecho/intel:**  All tests passed.

**derehco/gnu:** All tests passed, including snapshot tests for Kessler
and Held-Suarez

.github/scripts/pr_mod_file_tests.py

@@ -197,10 +197,10 @@ def _main_prog():
         #PR itself, so don't check its file type:
         if os.path.exists(file_obj.filename):
 
-            #Don't analyze 'manage_externals' files,
+            #Don't analyze 'git-fleximod' files,
             #as they are an external repo and thus
             #not our responsibility:
-            if 'manage_externals' not in file_obj.filename:
+            if 'git-fleximod' not in file_obj.filename:
 
                 #Check if it is a python file:
                 if _file_is_python(file_obj.filename):

.github/workflows/python_unit_tests.yml

@@ -44,8 +44,8 @@ jobs:
     # Run "checkout_externals":
     - name: Checkout externals
       run: |
-        manage_externals/checkout_externals --externals Externals_CAM.cfg ccpp-framework
-        manage_externals/checkout_externals --externals Externals.cfg cime
+        bin/git-fleximod update ccpp-framework
+        bin/git-fleximod update cime
     # Run python unit and doctests:
     - name: python unit tests
       run: |

.gitignore

@@ -1,20 +1,3 @@
-# Ignore externals
-ccpp_framework
-ccs_config
-chem_proc
-cime
-components
-libraries
-manage_externals.log
-share
-src/dynamics/mpas/dycore
-src/physics/carma/base
-src/physics/clubb
-src/physics/cosp2/src
-src/physics/ncar_ccpp
-src/physics/silhs
-src/utils/history_output
-
 # Ignore compiled python
 *.pyc
 buildnmlc

.gitmodules

@@ -0,0 +1,103 @@
+[submodule "ccpp-framework"]
+        path          = ccpp_framework
+        url           = https://github.com/peverwhee/ccpp-framework
+        fxtag         = CPF_0.2.056
+        fxrequired    = AlwaysRequired
+        fxDONOTUSEurl = https://github.com/peverwhee/ccpp-framework
+[submodule "mpas"]
+        path          = src/dynamics/mpas/dycore
+        url           = https://github.com/MPAS-Dev/MPAS-Model.git
+        fxsparse      = ../.mpas_sparse_checkout
+        fxtag         = ed6f8e39ec0a811b6d079ca0fc6f9fb6e30bad23
+        fxrequired    = AlwaysRequired
+        fxDONOTUSEurl = https://github.com/MPAS-Dev/MPAS-Model.git
+[submodule "ncar-physics"]
+        path          = src/physics/ncar_ccpp
+        url           = https://github.com/ESCOMP/atmospheric_physics
+        fxtag         = atmos_phys0_02_006
+        fxrequired    = AlwaysRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/atmospheric_physics
+[submodule "ccs_config"]
+        path          = ccs_config
+        url           = https://github.com/ESMCI/ccs_config_cesm.git
+        fxtag         = ccs_config_cesm0.0.106
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESMCI/ccs_config_cesm.git
+[submodule "cdeps"]
+        path          = components/cdeps
+        url           = https://github.com/ESCOMP/CDEPS.git
+        fxtag         = cdeps1.0.34
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/CDEPS.git
+[submodule "cice"]
+        path          = components/cice
+        url           = https://github.com/ESCOMP/CESM_CICE
+        fxtag         = cesm_cice6_5_0_10
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/CESM_CICE
+[submodule "cime"]
+        path          = cime
+        url           = https://github.com/ESMCI/cime
+        fxtag         = cime6.0.246
+        fxrequired    = ToplevelRequired
+	fxDONOTUSEurl = https://github.com/ESMCI/cime
+[submodule "cism"]
+        path          = components/cism
+        url           = https://github.com/ESCOMP/CISM-wrapper
+        fxtag         = cismwrap_2_2_001
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/CISM-wrapper
+[submodule "clm"]
+        path          = components/clm
+        url           = https://github.com/ESCOMP/CTSM
+        fxtag         = ctsm5.2.007
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/CTSM
+[submodule "cmeps"]
+        path          = components/cmeps
+        url           = https://github.com/ESCOMP/CMEPS.git
+        fxtag         = cmeps0.14.67
+        fxrequired    = ToplevelRequired
+	fxDONOTUSEurl = https://github.com/ESCOMP/CMEPS.git
+[submodule "fms"]
+        path          = libraries/FMS
+        url           = https://github.com/ESCOMP/FMS_interface
+        fxtag         = fi_240516
+        fxrequired    = ToplevelRequired
+	fxDONOTUSEurl = https://github.com/ESCOMP/FMS_interface
+[submodule "mct"]
+        path          = libraries/mct
+        url           = https://github.com/MCSclimate/MCT
+        fxtag         = MCT_2.11.0
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/MCSclimate/MCT
+[submodule "mizuRoute"]
+        path          = components/mizuRoute
+        url           = https://github.com/ESCOMP/mizuRoute
+        fxtag         = cesm-coupling.n02_v2.1.2
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/mizuRoute
+[submodule "mosart"]
+        path          = components/mosart
+        url           = https://github.com/ESCOMP/MOSART
+        fxtag         = mosart1_0_49
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/MOSART
+[submodule "parallelio"]
+        path          = libraries/parallelio
+        url           = https://github.com/NCAR/ParallelIO
+        fxtag         = pio2_6_2
+        fxrequired    = ToplevelRequired
+	fxDONOTUSEurl = https://github.com/NCAR/ParallelIO
+[submodule "rtm"]
+        path          = components/rtm
+        url           = https://github.com/ESCOMP/RTM
+        fxtag         = rtm1_0_79
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/RTM
+[submodule "share"]
+        path          = share
+        url           = https://github.com/ESCOMP/CESM_share
+        fxtag         = share1.0.19
+        fxrequired    = ToplevelRequired
+        fxDONOTUSEurl = https://github.com/ESCOMP/CESM_share

.lib/git-fleximod/.github/workflows/pre-commit

@@ -0,0 +1,13 @@
+name: pre-commit
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v3
+    - uses: pre-commit/[email protected]

.lib/git-fleximod/.github/workflows/pytest.yaml

@@ -0,0 +1,77 @@
+# Run this job on pushes to `main`, and for pull requests. If you don't specify
+# `branches: [main], then this actions runs _twice_ on pull requests, which is
+# annoying.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      # If you wanted to use multiple Python versions, you'd have specify a matrix in the job and
+      # reference the matrixe python version here.
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.9'
+
+      # Cache the installation of Poetry itself, e.g. the next step. This prevents the workflow
+      # from installing Poetry every time, which can be slow. Note the use of the Poetry version
+      # number in the cache key, and the "-0" suffix: this allows you to invalidate the cache
+      # manually if/when you want to upgrade Poetry, or if something goes wrong. This could be
+      # mildly cleaner by using an environment variable, but I don't really care.
+      - name: cache poetry install
+        uses: actions/cache@v4
+        with:
+          path: ~/.local
+          key: poetry-1.7.1
+
+      # Install Poetry. You could do this manually, or there are several actions that do this.
+      # `snok/install-poetry` seems to be minimal yet complete, and really just calls out to
+      # Poetry's default install script, which feels correct. I pin the Poetry version here
+      # because Poetry does occasionally change APIs between versions and I don't want my
+      # actions to break if it does.
+      #
+      # The key configuration value here is `virtualenvs-in-project: true`: this creates the
+      # venv as a `.venv` in your testing directory, which allows the next step to easily
+      # cache it.
+      - uses: snok/install-poetry@v1
+        with:
+          version: 1.7.1
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+
+      # Cache your dependencies (i.e. all the stuff in your `pyproject.toml`). Note the cache
+      # key: if you're using multiple Python versions, or multiple OSes, you'd need to include
+      # them in the cache key. I'm not, so it can be simple and just depend on the poetry.lock.
+      - name: cache deps
+        id: cache-deps
+        uses: actions/cache@v4
+        with:
+          path: .venv
+          key: pydeps-${{ hashFiles('**/poetry.lock') }}
+
+      # Install dependencies. `--no-root` means "install all dependencies but not the project
+      # itself", which is what you want to avoid caching _your_ code. The `if` statement
+      # ensures this only runs on a cache miss.
+      - run: poetry install --no-interaction --no-root
+        if: steps.cache-deps.outputs.cache-hit != 'true'
+
+      # Now install _your_ project. This isn't necessary for many types of projects -- particularly
+      # things like Django apps don't need this. But it's a good idea since it fully-exercises the
+      # pyproject.toml and makes that if you add things like console-scripts at some point that
+      # they'll be installed and working.
+      - run: poetry install --no-interaction
+
+      # And finally run tests. I'm using pytest and all my pytest config is in my `pyproject.toml`
+      # so this line is super-simple. But it could be as complex as you need.
+      - run: |
+          git config --global user.name "${GITHUB_ACTOR}"
+          git config --global user.email "${GITHUB_ACTOR_ID}+${GITHUB_ACTOR}@users.noreply.github.com"
+          poetry run pytest
+

.lib/git-fleximod/.pre-commit-config.yaml

@@ -0,0 +1,18 @@
+exclude: ^utils/.*$
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+  - repo: https://github.com/psf/black
+    rev: 22.3.0
+    hooks:
+      - id: black
+  - repo: https://github.com/PyCQA/pylint
+    rev: v2.11.1
+    hooks:
+      - id: pylint
+        args:
+          - --disable=I,C,R,logging-not-lazy,wildcard-import,unused-wildcard-import,fixme,broad-except,bare-except,eval-used,exec-used,global-statement,logging-format-interpolation,no-name-in-module,arguments-renamed,unspecified-encoding,protected-access,import-error,no-member

.lib/git-fleximod/CODE_OF_CONDUCT.md

@@ -0,0 +1,107 @@
+# Contributor Code of Conduct
+_The Contributor Code of Conduct is for participants in our software projects and community._
+
+## Our Pledge
+We, as contributors, creators, stewards, and maintainers (participants), of **git-fleximod** pledge to make participation in
+our software, system or hardware project and community a safe, productive, welcoming and inclusive experience for everyone.
+All participants are required to abide by this Code of Conduct.
+This includes respectful treatment of everyone regardless of age, body size, disability, ethnicity, gender identity or expression,
+level of experience, nationality, political affiliation, veteran status, pregnancy, genetic information, physical appearance, race,
+religion, or sexual orientation, as well as any other characteristic protected under applicable US federal or state law.
+
+## Our Standards
+Examples of behaviors that contribute to a positive environment include:
+
+* All participants are treated with respect and consideration, valuing a diversity of views and opinions
+* Be considerate, respectful, and collaborative
+* Communicate openly with respect for others, critiquing ideas rather than individuals and gracefully accepting criticism
+* Acknowledging the contributions of others
+* Avoid personal attacks directed toward other participants
+* Be mindful of your surroundings and of your fellow participants 
+* Alert UCAR staff and suppliers/vendors if you notice a dangerous situation or someone in distress
+* Respect the rules and policies of the project and venue
+
+Examples of unacceptable behavior include, but are not limited to:
+
+* Harassment, intimidation, or discrimination in any form
+* Physical, verbal, or written abuse by anyone to anyone, including repeated use of pronouns other than those requested
+* Unwelcome sexual attention or advances
+* Personal attacks directed at other guests, members, participants, etc.
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Alarming, intimidating, threatening, or hostile comments or conduct
+* Inappropriate use of nudity and/or sexual images 
+* Threatening or stalking anyone, including a participant
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Scope
+This Code of Conduct applies to all spaces managed by the Project whether they be physical, online or face-to-face.
+This includes project code, code repository, associated web pages, documentation, mailing lists, project websites and wiki pages,
+issue tracker, meetings, telecons, events, project social media accounts, and any other forums created by the project team which the
+community uses for communication.
+In addition, violations of this Code of Conduct outside these spaces may affect a person's ability to participate within them.
+Representation of a project may be further defined and clarified by project maintainers.
+
+## Community Responsibilities
+Everyone in the community is empowered to respond to people who are showing unacceptable behavior.
+They can talk to them privately or publicly.
+Anyone requested to stop unacceptable behavior is expected to comply immediately.
+If the behavior continues concerns may be brought to the project administrators or to any other party listed in the
+[Reporting](#reporting) section below.
+
+## Project Administrator Responsibilities
+Project administrators are responsible for clarifying the standards of acceptable behavior and are encouraged to model appropriate
+behavior and provide support when people in the community point out inappropriate behavior.
+Project administrator(s) are normally the ones that would be tasked to carry out the actions in the [Consequences](#consequences)
+section below.
+
+Project administrators are also expected to keep this Code of Conduct updated with the main one housed at UCAR, as listed below in
+the [Attribution](#attribution) section.
+
+## Reporting
+Instances of unacceptable behavior can be brought to the attention of the project administrator(s) who may take any action as
+outlined in the [Consequences](#consequences) section below.
+However, making a report to a project administrator is not considered an 'official report' to UCAR. 
+
+Instances of unacceptable behavior may also be reported directly to UCAR pursuant to [UCAR's Harassment Reporting and Complaint
+Procedure](https://www2.fin.ucar.edu/procedures/hr/harassment-reporting-and-complaint-procedure), or anonymously through [UCAR's
+EthicsPoint Hotline](https://www2.fin.ucar.edu/ethics/anonymous-reporting).
+
+Complaints received by UCAR will be handled pursuant to the procedures outlined in UCAR's Harassment Reporting and Complaint
+Procedure.
+Complaints to UCAR will be held as confidential as practicable under the circumstances, and retaliation against a person who
+initiates a complaint or an inquiry about inappropriate behavior will not be tolerated.
+
+Any Contributor can use these reporting methods even if they are not directly affiliated with UCAR.
+The Frequently Asked Questions (FAQ) page for reporting is [here](https://www2.fin.ucar.edu/procedures/hr/reporting-faqs).
+
+## Consequences
+Upon receipt of a complaint, the project administrator(s) may take any action deemed necessary and appropriate under the
+circumstances.
+Such action can include things such as: removing, editing, or rejecting comments, commits, code, wiki edits, email, issues, and
+other contributions that are not aligned to this Code of Conduct, or banning temporarily or permanently any contributor for other
+behaviors that are deemed inappropriate, threatening, offensive, or harmful.
+Project administrators also have the right to report violations to UCAR HR and/or UCAR's Office of Diversity, Equity and Inclusion
+(ODEI), as well as a participant's home institution and/or law enforcement.
+In the event an incident is reported to UCAR, UCAR will follow its Harassment Reporting and Complaint Procedure.
+
+## Process for Changes
+All UCAR managed projects are required to adopt this Contributor Code of Conduct.
+Adoption is assumed even if not expressly stated in the repository.
+Projects should fill in sections where prompted with project-specific information, including, project name and adoption date.
+
+Projects that adopt this Code of Conduct need to stay up to date with UCAR's Contributor Code of Conduct, linked with a DOI in the
+[Attribution](#attribution) section below.
+Projects can make limited substantive changes to the Code of Conduct, however, the changes must be limited in scope and may not
+contradict the UCAR Contributor Code of Conduct.
+
+## Attribution
+This Code of Conduct was originally adapted from the [Contributor Covenant](http://contributor-covenant.org/version/1/4), version
+1.4.
+We then aligned it with the UCAR Participant Code of Conduct, which also borrows from the American Geophysical Union (AGU) Code of
+Conduct.
+The UCAR Participant Code of Conduct applies to both UCAR employees as well as participants in activities run by UCAR.
+The original version of this for all software projects that have strong management from UCAR or UCAR staff is available on the UCAR
+website at https://doi.org/10.5065/6w2c-a132.
+The date that it was adopted by this project was **Feb/13/2018**.
+When responding to complaints, UCAR HR and ODEI will do so based on the latest published version.
+Therefore, any project-specific changes should follow the [Process for Changes](#process-for-changes) section above.

.lib/git-fleximod/License

@@ -0,0 +1,20 @@
+Copyright 2024 NSF National Center for Atmospheric Sciences (NCAR)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+“Software”), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.