Making code changes in the UFS weather model and its subcomponents

UFS-Weather-Model Code Development

Introduction

The development process for the UFS-Weather-Model has been structured to make best use of the tools git/github provides for code development. Here is a brief summary of whats included in this page:

• Intro
  • Repository Structure
  • Submodules
• First step
  • Creating a GitHub Issue for feature or bugfix
• Obtaining code and committing changes
  • Forking
  • Cloning and setup
  • Committing
• Developing in subcomponents
  • Bringing in your changes from subcomponents
• Regression Testing
  • Steps for proper regression testing
• Pushing and PR's
  • Setting up remotes
  • Pushing Changes
  • Creating a PR
    • Before making a PR
    • Creating a PR
    • Code review
• Specifics for NOAA Staff

Intro

Build structure and submodules

The UFS-Weather-Model uses a CMake build environment which parallels well with the hierarchical code structure used in the repository.

UFS-Weather-Model Repository Structure:

• Model build files:
  • CMakeLists.txt -- Top level build instructions for CMake.
  • cmake/ -- Files used by CMake in building UFS-Weather-Model.
• System setup:
  • conf/ -- model specific flag and variables for multiple machine types.
  • modulefiles/ -- machine specific variables for build.
• Testing:
  • tests/* -- Regression testing files.
• GIT specific files:
  • .gitmodules -- Defines the submodules needed for proper build of UFS-Weather-Model.

UFS-Weather-Model Submodules

• DATM
• FMS
• NEMS
  • NCEPLIBS-pyprodutil
• fv3atm
  • GFDL_atmos_cubed_sphere
  • ccpp-framework
  • ccpp-physics
• stochastic_physics
• WW3
• MOM6-interface
  • MOM6
• CICE-interface
  • CICE6
• CMEPS-interface
  • CMEPS

Notes

• The model and its subcomponents are housed in authoritative (official) repositories. The UFS-Weather-Model uses git submodules to manage its subcomponents.
• All the authoritative repositories are read-only for users.
• Branches under the authoritative repositories include main develop (master) branch, production branches, and public release branches.
• Branches under the authoritative repositories are protected.

First step

All development on the UFS-Weather-Model repository MUST have a GitHub Issue created. This allows the code managers and the community to discuss the importance and timeline as it fits best into our development cycle. It is possible to fix multiple issues with a single Pull Request (PR) which is why every PR must be linked with at least one issue

Obtaining code (Using UFS-Weather-Model repository as an example)

Before you start:

*If you do not have a GitHub account, please get one at github.org. EMC staff can also refer to Kate Friedman’s document: “EMC GitHub Transition”:

• Set up a Github account
• Get added to the NOAA-EMC org people list

Forking the authoritative repository

• A fork is a copy of a repository. You manage your own fork.
• Forks let you make changes to a project without affecting the original repository.
• You can fetch updates from the original repository.
• You can submit changes to the original repository with PR's.
• For more details, please see Working with forks.

Cloning and setup of local repository

• Create a fork via github:
  • Login to your personal GitHub account
  • Go to the official code repository website (e.g. https://github.com/ufs-community/ufs-weather-model), click on “fork” on the top right corner. You will see the repository in your own GitHub account.
• Cloning your fork:

  git clone https://github.com/<your_github_username>/ufs-weather-model
  cd ufs-weather-model

• Checking out the develop branch:

  git checkout develop

  • NOTE: You can check which branch you are using at any time by typing git branch in the top level of your repository.
• Retrieve submodule files:

  git submodule update --init --recursive

  • NOTE: --init will initialize any submodule you have not already initialized.
  • NOTE: --recursive will make sure to get submodules inside the submodules listed in your .gitmodules file.

Making changes & Committing

• Creating a feature branch to make changes in:

  git checkout -b feature/newfeaturename

  • NOTE: We typically use feature and bugfix (i.e. feature/addfunction or bugfix/missingtags)
  • You can confirm your using the new branch name by typing git branch in the top level of your repository
• Make your change or edit or fix.
• After each change or edit or fix:

  git commit -m "Type an explanation of the change or edit or fix here"

  • NOTE: Committing after each change or edit or fix gives the code maintainers a very detailed understanding of what you're doing after you submit your PR.

Developing in subcomponents

While developing in subcomponents, the process of forking into your account, cloning it, committing and pushing is identical to what is represented here. The steps that follow pertain to what needs to be done inside your forked UFS-Weather-Model reposotiry to bring in the changes you have made to subcomponents.

Bringing in your changes from subcomponents

• If you haven't cloned your fork yet using git clone:
  • On your fork in GitHub edit your .gitmodules file.
  • If you made changes in fv3atm that you want to bring in to test in UFS-Weather-Model you would edit:

    [submodule "FV3"]
    path = FV3
    url = https://github.com/NOAA-EMC/fv3atm
    branch = develop
    

  • such that you comment out the url and branch lines and add the appropriate information from your personal fork of fv3atm to look like this:

    [submodule "FV3"]
    path = FV3
    #url = https://github.com/NOAA-EMC/fv3atm
    #branch = develop
    url = https://github.com/<your_github_username>/fv3atm
    branch = <your branch name>
    

  • then clone, checkout and submodule update as previously instructed.
• If you have already used clone and submodule update locally:
  • You should update your .gitmodules file as instructed above.
  • cd into your FV3: cd FV3
  • update your remotes

    git remote rename origin upstream
    git remote add origin https://github.com/<your_github_username>/fv3atm
    

  • checkout your branch

    git fetch origin
    git checkout origin/<your_branch_name>
    

    • NOTE: You can verify your branch by typing in git branch.

Regression Testing

• Regression testing is your way to prove to us that the changes you are proposing through a PR do not change the build or operation of the model in a negative way. When development work is done, users need to sync their feature branch with the latest develop/master branch in official repository and run the regression tests:

  setenv NEMS_COMPILER GNU
  cd tests
  ./rt.sh -fe -l rt_gnu.conf
  

  • NOTE: This example is for those that are using the GNU compiler.
• For an in-depth tutorial on Regression Testing please see the Regression Testing section.

Pushing code and PR's

Once all of your commits have been completed and you've passed all the regression tests (if applicable) you will need to push those changes to your fork on github inside the branch you created

Setting up remotes

1. Check to see if you already have an origin and upstream remote location set up

   git remote -v

   • You should see something like the following if it is already setup correctly:

   upstream        https://github.com/ufs-community/ufs-weather-model (fetch)  
   upstream        https://github.com/ufs-community/ufs-weather-model (push)  
   origin  https://github.com/your_github_account/ufs-weather-model (fetch)  
   origin  https://github.com/your_github_account/ufs-weather-model (push)

2. If you are missing upstream then you need to setup your remotes to add the authoritative repository:

   git remote add upstream https://github.com/ufs-community/ufs-weather-model

3. Last you should check again to make sure your remotes look like above.

Pushing Changes

• Push all your commits to your fork.

  git push origin feature/newfeaturename

  • NOTE: Make sure you choose the remote of origin as that is where your personal fork of the authoritative repository is.
  • NOTE: Make sure to match the branch name you created earlier (i.e. feature/newfeaturename).

Pull Requests

A PR tells the code managers that your code changes from your feature branch are ready to be merged into the develop branch in the official repository. Users need to make a PR for each of the repository that you changed in the .gitmodule. You can start from the subcomponent repository, then make PR to each repository up to the UFS-Weather-Model application repository

Before making a PR

• Make sure that an Issue has been created.
• The users feature branch should be merged with the corresponding branch in the authoritative repository with all of the conflicts resolved.
• Regression test needs to be passed on all of the supported platforms.
  • Code manager(s) will run the regression test on the required platforms that users don’t have access to. *If a user needs a test run on a system they do not have access to, then please note this in a draft PR. Once all tests pass, then the PR may be submitted for review.

Creating a PR

1. Go to the your_github_account repository, choose your branch, then click on “new pull request”.
2. In the next page, choose base repository and base branch. By default, PR's are based on the parent repository default branch.
3. Provide a Title
4. Fill out the template provided
5. Click on “Create Pull Request” button.

Code review

Reviews allow collaborators to comment on the changes proposed in PR's, approve the changes, or request further changes before the PR is merged.

• Code managers will add reviewers.
• At least one reviewer with write permission needs to give approval before the code can be committed.
• Anyone with read access can review and comment on the changes it proposes.

Specifics for NOAA staff

• Add specific configurations or setup steps for NOAA staff that use NOAA HPC systems

Questions/suggestions

Please create an Issue if you have any questions/suggestions.