Replace checkout script with submodules (#2142)

Replaces the checkout script with git submodules. This change brings workflow in
line with other repos and git standards.

Due to these changes, the build system is updated slightly. First, the options that
were formerly used with `checkout.sh` to reduce unneeded builds (`-g` for GSI
and `-u` for UFSDA) are moved to `build_all.sh` instead. This is needed since build
all will not be able to key off directories existing or not to determine which com-
ponents to build. Second, the partial build system is removed. It was rarely, if ever,
used, and parts of it were already ignored with the previous changes to base builds
off whether directories exist. The manage externals file is also removed since that is
also not being used.

Following these changes, clones/checkout will need to either be recursive, or 
`git submodule update` will need to be called separately to make sure components
are properly checked out (non-exhaustive list):

```
## Checkout develop (method 1)
git clone --recursive https://github.com/NOAA-EMC/global-workflow destination

## Checkout develop (method 2)
git clone  --jobs 8 https://github.com/NOAA-EMC/global-workflow destination
cd destination
git submodule update --init --recursive --jobs 8

## Checkout existing branch some_branch (method 1)
git clone --recursive  --jobs 8 -b some_branch https://github.com/NOAA-EMC/global-workflow destination

## Checkout existing branch some_branch (method 2)
git clone https://github.com/NOAA-EMC/global-workflow destination
cd destination
git checkout --recurse-submodules some_branch

## Checkout existing branch some_branch (method 3)
git clone https://github.com/NOAA-EMC/global-workflow destination
cd destination
git checkout some_branch
git submodule update --init --recursive  --jobs 8

## Checkout new branch new_branch
git clone https://github.com/NOAA-EMC/global-workflow destination
cd destination
git checkout -b new_branch
git submodule update --init --recursive  --jobs 8
```

The git ignore list had to be updated to use explicit paths as the wildcard paths pick
up the ones that are now submodules.

UFS creates files during the build process that are not in its ignore list, so that sub-
module ignores the "dirty" state to avoid confusing users running `git status`.

CI and documentation are also updated to reflect the changes.

Resolves #2141

.gitignore

@@ -31,6 +31,7 @@ fix/cice
 fix/cpl
 fix/datm
 fix/gldas
+fix/gdas
 fix/gsi
 fix/lut
 fix/mom6
@@ -40,7 +41,6 @@ fix/sfc_climo
 fix/ugwd
 fix/verif
 fix/wave
-fix/wafs
 
 # Ignore parm file symlinks
 #--------------------------
@@ -97,8 +97,42 @@ parm/wafs
 #--------------------------------------------
 sorc/*log
 sorc/logs
-sorc/*.cd
-sorc/*.fd
+sorc/calc_analysis.fd
+sorc/calc_increment_ens.fd
+sorc/calc_increment_ens_ncio.fd
+sorc/emcsfc_ice_blend.fd
+sorc/emcsfc_snow2mdl.fd
+sorc/enkf.fd
+sorc/enkf_chgres_recenter_nc.fd
+sorc/fbwndgfs.fd
+sorc/gaussian_sfcanl.fd
+sorc/getsfcensmeanp.fd
+sorc/getsigensmeanp_smooth.fd
+sorc/getsigensstatp.fd
+sorc/gfs_bufr.fd
+sorc/global_cycle.fd
+sorc/gsi.fd
+sorc/interp_inc.fd
+sorc/mkgfsawps.fd
+sorc/overgridid.fd
+sorc/oznmon_horiz.fd
+sorc/oznmon_time.fd
+sorc/radmon_angle.fd
+sorc/radmon_bcoef.fd
+sorc/radmon_bcor.fd
+sorc/radmon_time.fd
+sorc/rdbfmsua.fd
+sorc/recentersigp.fd
+sorc/reg2grb2.fd
+sorc/supvit.fd
+sorc/syndat_getjtbul.fd
+sorc/syndat_maksynrc.fd
+sorc/syndat_qctropcy.fd
+sorc/tave.fd
+sorc/tocsbufr.fd
+sorc/upp.fd
+sorc/vint.fd
+sorc/webtitle.fd
 
 # Ignore scripts from externals
 #------------------------------
@@ -121,6 +155,7 @@ scripts/exgfs_atmos_wafs_grib2_0p25.sh
 ush/chgres_cube.sh
 ush/emcsfc_ice_blend.sh
 ush/emcsfc_snow.sh
+ush/exglobal_prep_ocean_obs.py
 ush/fix_precip.sh
 ush/fv3gfs_driver_grid.sh
 ush/fv3gfs_filter_topo.sh
@@ -150,7 +185,6 @@ versions/build.ver
 versions/run.ver
 
 # wxflow checkout and symlinks
-sorc/wxflow
 ush/python/wxflow
 workflow/wxflow
 ci/scripts/wxflow

.gitmodules

@@ -0,0 +1,28 @@
+[submodule "sorc/ufs_model.fd"]
+	path = sorc/ufs_model.fd
+	url = https://github.com/ufs-community/ufs-weather-model
+	ignore = dirty
+[submodule "sorc/wxflow"]
+	path = sorc/wxflow
+	url = https://github.com/NOAA-EMC/wxflow
+[submodule "sorc/gfs_utils.fd"]
+	path = sorc/gfs_utils.fd
+	url = https://github.com/NOAA-EMC/gfs-utils
+[submodule "sorc/ufs_utils.fd"]
+	path = sorc/ufs_utils.fd
+	url = https://github.com/ufs-community/UFS_UTILS.git
+[submodule "sorc/verif-global.fd"]
+	path = sorc/verif-global.fd
+	url = https://github.com/NOAA-EMC/EMC_verif-global.git
+[submodule "sorc/gsi_enkf.fd"]
+	path = sorc/gsi_enkf.fd
+	url = https://github.com/NOAA-EMC/GSI.git
+[submodule "sorc/gdas.cd"]
+	path = sorc/gdas.cd
+	url = https://github.com/NOAA-EMC/GDASApp.git
+[submodule "sorc/gsi_utils.fd"]
+	path = sorc/gsi_utils.fd
+	url = https://github.com/NOAA-EMC/GSI-Utils.git
+[submodule "sorc/gsi_monitor.fd"]
+	path = sorc/gsi_monitor.fd
+	url = https://github.com/NOAA-EMC/GSI-Monitor.git

ci/scripts/clone-build_ci.sh

@@ -48,7 +48,7 @@ git clone "${REPO_URL}"
 cd global-workflow || exit 1
 
 # checkout pull request
-"${GH}" pr checkout "${PR}" --repo "${REPO_URL}"
+"${GH}" pr checkout "${PR}" --repo "${REPO_URL}" --recurse-submodules
 HOMEgfs="${PWD}"
 source "${HOMEgfs}/ush/detect_machine.sh"
 
@@ -67,30 +67,14 @@ source "${HOMEgfs}/ush/detect_machine.sh"
 commit=$(git log --pretty=format:'%h' -n 1)
 echo "${commit}" > "../commit"
 
-# run checkout script
+# build full cycle
 cd sorc || exit 1
 set +e
-./checkout.sh -c -g -u >> log.checkout 2>&1
-checkout_status=$?
-DATE=$(date +'%D %r')
-if [[ ${checkout_status} != 0 ]]; then
-  {
-    echo "Checkout: *** FAILED ***"
-    echo "Checkout: Failed at ${DATE}"
-    echo "Checkout: see output at ${PWD}/log.checkout"
-  } >> "${outfile}"
-  exit "${checkout_status}"
-else
-  {
-    echo "Checkout: Completed at ${DATE}"
-  } >> "${outfile}"
-fi
 
-# build full cycle
 source "${HOMEgfs}/ush/module-setup.sh"
 export BUILD_JOBS=8
 rm -rf log.build
-./build_all.sh  >> log.build 2>&1
+./build_all.sh -gu  >> log.build 2>&1
 build_status=$?
 
 DATE=$(date +'%D %r')

ci/scripts/driver_weekly.sh

@@ -66,7 +66,7 @@ echo "Creating new branch ${branch} from develop on ${MACHINE_ID} in ${develop_d
 rm -Rf "${develop_dir}"
 mkdir -p "${develop_dir}"
 cd "${develop_dir}" || exit 1
-git clone "${REPO_URL}"
+git clone --recursive "${REPO_URL}"
 cd global-workflow || exit 1
 git checkout -b "${branch}"
 

docs/source/clone.rst

@@ -15,35 +15,34 @@ Clone the `global-workflow` and `cd` into the `sorc` directory:
 
 ::
 
-   git clone https://github.com/NOAA-EMC/global-workflow
+   git clone --recursive https://github.com/NOAA-EMC/global-workflow
    cd global-workflow/sorc
 
-For forecast-only (coupled or uncoupled) checkout the components:
+For forecast-only (coupled or uncoupled) build of the components:
 
 ::
 
-   ./checkout.sh
+   ./build_all.sh
 
-For cycled (w/ data assimilation) use the `-g` option during checkout:
+For cycled (w/ data assimilation) use the `-g` option during build:
 
 ::
 
-   ./checkout.sh -g
+   ./build_all.sh -g
 
-For coupled cycling (include new UFSDA) use the `-gu` options during checkout:
+For coupled cycling (include new UFSDA) use the `-gu` options during build:
 
 [Currently only available on Hera, Orion, and Hercules]
 
 ::
 
-   ./checkout.sh -gu
+   ./build_all.sh -gu
 
 
 Build workflow components and link workflow artifacts such as executables, etc.
 
 ::
 
-   ./build_all.sh
    ./link_workflow.sh
 
 
@@ -73,7 +72,7 @@ You now have a cloned copy of the global-workflow git repository. To checkout a
 
 ::
 
-   git checkout BRANCH_NAME
+   git checkout --recurse-submodules BRANCH_NAME
 
 .. note::
    Branch must already exist. If it does not you need to make a new branch using the ``-b`` flag:
@@ -86,62 +85,55 @@ The ``checkout`` command will checkout BRANCH_NAME and switch your clone to that
 
 ::
 
-   git checkout my_branch
+   git checkout --recurse-submodules my_branch
    git branch
    * my_branch
    develop
 
-**********
-Components
-**********
+Using ``--recurse-submodules`` is important to ensure you are updating the component versions to match the branch.
 
-Once you have cloned the workflow repository it's time to checkout/clone its components. The components will be checked out under the ``/sorc`` folder via a script called checkout.sh. Run the script with no arguments for forecast-only:
+^^^^^^^^^^^^^^^^
+Build components
+^^^^^^^^^^^^^^^^
 
-::
+Under the ``/sorc`` folder is a script to build all components called ``build_all.sh``. After checking out the branch you wish to use, run this script to build all components codes:
 
-   cd sorc
-   ./checkout.sh
+::
 
-Or with the ``-g`` switch to include data assimilation (GSI) for cycling:
+   ./build_all.sh [-a UFS_app][-g][-h][-u][-v]
+  -a UFS_app:
+    Build a specific UFS app instead of the default
+  -g:
+    Build GSI
+  -h:
+    Print this help message and exit
+  -j:
+    Specify maximum number of build jobs (n)
+  -u:
+    Build UFS-DA
+  -v:
+    Execute all build scripts with -v option to turn on verbose where supported
+
+For forecast-only (coupled or uncoupled) build of the components:
 
 ::
 
-   cd sorc
-   ./checkout.sh -g
+   ./build_all.sh
 
-Or also with the ``-u`` swtich to include coupled DA (via UFSDA):
-[Currently only available on Hera, Orion, and Hercules]
+For cycled (w/ data assimilation) use the `-g` option during build:
 
 ::
 
-   cd sorc
-   ./checkout.sh -gu
-
-Each component cloned via checkout.sh will have a log (``/sorc/logs/checkout-COMPONENT.log``). Check the screen output and logs for clone errors.
+   ./build_all.sh -g
 
-^^^^^^^^^^^^^^^^
-Build components
-^^^^^^^^^^^^^^^^
+For coupled cycling (include new UFSDA) use the `-gu` options during build:
 
-Under the ``/sorc`` folder is a script to build all components called ``build_all.sh``. After running checkout.sh run this script to build all components codes:
+[Currently only available on Hera, Orion, and Hercules]
 
 ::
 
-   ./build_all.sh [-a UFS_app][-c build_config][-h][-v]
-   -a UFS_app:
-   Build a specific UFS app instead of the default
-   -c build_config:
-   Selectively build based on the provided config instead of the default config
-   -h:
-   Print usage message and exit
-   -v:
-   Run all scripts in verbose mode
-
-A partial build option is also available via two methods:
-
-  a) modify gfs_build.cfg config file to disable/enable particular builds and then rerun build_all.sh
+   ./build_all.sh -gu
 
-  b) run individual build scripts also available in ``/sorc`` folder for each component or group of codes
 
 ^^^^^^^^^^^^^^^
 Link components