-
Notifications
You must be signed in to change notification settings - Fork 0
Developers Guide
- Table of Contents
- Roles and Responsibilities
- Important Concepts
- Testing Requirements (Developer)
- Workflows
- Other Concepts
Adding value to EarthWorks, but not currently acting in another role. Every role is at least a Contributor and must fulfill these responsibilities as well. Responsible for upholding Code of Conduct.
Has changes to submit to EarthWorks either directly to the model or to an external that eventually makes it way to a model. They are responsible for their changes throughout the PR process, it is their duty to follow workflows and recommendations from other collaborators until the PR is merged into the main model repository.
Has been engaged in the PR process, typically by the developer. They are concerned with the quality of the changes as it concerns the project. A reviewer is responsible for timely responses, change evaluation (to accept, reject, or modify) , and providing constructive feedback.
May also be a Reviewer of a PR. They are concerned with the Git and GitHub "presence" of EarthWorks repositories. It is their responsibility to maintain a helpful Git history of all repos (including branch structure, tags, descriptions, etc.), maintain a productive and sensible GitHub environment (branches allowed in main repo, Issues triage), collaborating on and creating "support documents" (Code_of_Conduct, Guides, workflows), and provide some user support (if it occurs). Within the context of a PR, the maintainer is responsible for adding the approved changes to the develop branch(es) of the involved repo(s).
Guides the overall EarthWorks project. They are responsible for forward looking objectives, supporting development, fostering a community (eventually), and communication about the project. They are also responsible for mediating any conflict and having "final say" when a decision is needed.
As mentioned the "main model repository" is EarthWorksOrg/EarthWorks. The main purpose of this repo is to collect all of the related "top-level" configurations, certain meta-documents (e.g. LICENSE) for the project, and to provide a list of "externals" with a mechanism to fetch them.
==? Directory picture ?==
An external in this case refers to a codebase that isn't integrated with the one being considered. For EarthWorks, externals include codebases like ccs_configs, CAM, CAM's externals, and MOSART. Like CESM, we use the manage_externals/checkout_externals tool to fetch these dependencies after cloning a fresh copy of the EarthWorks model. In later parts of this guide, any repo that contains an external is an "over-repo" of that external (e.g. CAM is an over-repo of CLUBB).
The externals EarthWorks uses are defined in the Externals.cfg file. A specific version of the external is fetched according to the URL and hash, branch, or tag provided.
Further, some of these externals have been forked into EarthWorksOrg versions - we call these "EarthWorks externals" or "EW externals." These repos were forked so that EarthWorks development can move at its own pace and not be held to the pace and all the practices of the repositories we forked from (i.e. the "upstream" repositories). These forks also allow for the EarthWorks team to pursue our own development that may not be desired in the upstreams.
There are also three EW externals that are wholly unique: EarthWorksOrg/mpas-framework, EarthWorksOrg/mpas-ocean, and EarthWorksOrg/mpas-seaice. These EW externals were based on work in the E3SM v2.0 code ==(?)== and have no traditional upstream repositories.
==? List of EWOrg repos? Pointer to EarthWorksOrg/EarthWorks/repositories page?==
EarthWorks uses annotated tags in each of our repos to While the checkout_externals script can work with a branch, a tag, or a hash, a tag is preferred for use in the various EarthWorks repos' main and development branches. A branch can change without warning, particularly if it is from someone's personal fork. A hash is somewhat esoteric (non-human readable) but is very stable. An annotated tag gives the best of both: human readable, semi-stable reference points that contain some of their own documentation.
The overall format of our tags are in an adapted Semantic Versioning format. They are generally of the form {pre}{X}.{Y}.{Z}, where pre stands for some repo- or tag-specific prefix, X is the major version, Y is the minor verion, and Z is the patch version. So far, the patch number has been zero-padded to three digits. Example tag: cam-ew2.1.002
Also note that in EarthWorks we make a distinction between "development" tags and "release" tags. Development tags almost always occur on the development branches (across EW repos) and are in an adapted Semantic Versioning format. They are generally of the form {pre}-ew{X}.{Y}.{Z}, where pre stands for some repo- or tag-specific prefix, X is the major version, Y is the minor verion, and Z is the patch version. So far, the patch number has been zero-padded to three digits. Example tag: cam-ew2.1.002. Release tags are similar, only occur on the main branches (across EW repos), and typically fit the format release-{pre}-ew{X}.{Y}. The patch number is dropped since most releases should only be for major and minor versions. In exceptional circumstances a "bug fix" or "patch release" may occur and would fit the format release-{pre}{X}.{Y}.{Z}.
Development tags in EarthWorks have the prefix set to ewm- and it is typically sufficient to provide only a short (even one-line) description of the changes being tagged.
Example:
> git show ewm-2.1.002
tag ewm-2.1.002
Tagger: G. Dylan Dickerson <[email protected]>
Date: Tue Apr 16 11:01:09 2024 -0600
Fix GPU builds for compsets with multiple MPAS cores (PR #39)
commit 4b974c57bdb6544acddf0647f1598d6fefe0feb8 (tag: ewm-2.1.002)
Merge: 258a4d4 24948ad
Author: G. Dylan Dickerson <[email protected]>
Date: Tue Apr 2 14:53:20 2024 -0600
Merge branch 'framework-ext-refs' into develop (PR #39)
Add changes in mpas-framework, mpas-seaice, and mpas-ocean to ensure
unique module names across the different copies of the MPAS
infrastructure (framework, operators, etc) that are compiled for each
MPAS core.Development tags in EW externals have prefixes according to their repo name and the tag's description follows a specific format.
Besides the summary and description, the last line of any tag in an EW external must end with a line that describes the last point in common with the upstream repo. The format for the last line looks like Last changes from upstream '{UPSTREAM_OWNER/UPSTREAM_NAME}' tag:'{UPSTREAM_TAG}'. This line can be carried over from the previous tag, unless an "upstream update" is being done.
? Table of Repo and Prefix here?
Example
> git show cam-ew2.1.002
tag cam-ew2.1.002
Tagger: G. Dylan Dickerson <[email protected]>
Date: Wed Mar 13 16:01:59 2024 -0600
Fix intel-oneapi builds
Last changes from upstream 'ESCOMP/CAM' tag:'cam6_3_148'
commit d5e3529e2ee4e61375d00aa5485a14fe8934ab8e (HEAD, tag: cam-ew2.1.002, origin/ew-develop)
Merge: f09ed7cd 3e9870d9
Author: G. Dylan Dickerson <[email protected]>
Date: Wed Mar 13 16:01:46 2024 -0600
Merge branch 'gdicker1/fix/perllogic_intel-oneapi' into ew-develop (PR #15)
Fix ability to build CAM with intel-oneapi compilers.These are special cases of the above tagging schemes. Release tags are intended for "general users" and non-developers and have this format to make them recognizable (e.g. in error logs).
For the EarthWorks model repo, release tags typically look like: release-ew{X}.{Y}.
An example of a release tag from an EW external looks like: release-cam-ew2.1. Please remember that all tags in an EW external must describe the last tag merged from the upstream repo.
As you develop, you should have been testing exercising your code changes along the way. This is the necessary first step for any change to be submitted.
Then, most PRs will require the "Minimal Testing." Minimal Testing includes successful builds and runs of
- GPU run of FullyCoupled model on 120km grid for 1 day (NVIDIA compiler only)
- (If GPU not possible) CPU run of FullyCoupled model on 120km (All supported compilers?)
- Any other tests the developer (you) mentions
A release or some other changes may require Verification Testing (at discretion of Reviewer(s) and Leadership). Verification Testing includes:
- ==... ? Coming Soon==
If you have some changes to submit, the zeroth step is figuring out which repository your changes go into. If you are unsure, start talking with other EarthWorks Developers to determine which repository your changes belong in. The rest of this section assumes your changes will go into an EarthWorksOrg fork of the repo, and refers to this repo as {R}.
Rarely, if your change are only specific to the main model repository, you can just open a PR in the main model repository. Examples of such changes include editing the meta documents like README.md or changing an Externals.cfg entry to use a new version (especially for non-EW externals).
Otherwise there are two other questions that help guide the process: Q1. Is this change EarthWorks-specific? Q2. If no, what is the urgency of this change?
If the answer to the first question is "yes," then the process involves:
- basing your branch off of develop in the EWOrg/
{R}repository - creating a PR in EWOrg/
{R} - creating branches and PRs in the over-repos that use these changes
- NOTE: this step can help ensure that others can easily fetch a sandbox with the changes you wish to have tested
- working through the PR process with other EarthWorks Contributors
- once approved, working with the Maintainer to ensure your changes "run up the chain" of repos.
However, if the answer to the first question is "no," then the process involves:
- basing your branch off of the last "upstream ref" on the develop branch in EWOrg/
{R}
- NOTE: this information should be in the most recent EarthWorks tag for your branch and will involve adding another remote to your local repository
- creating a PR in EWOrg/
{R} - creating a PR in the upstream repo of
{R}and following their PR process - making a decision based on the answer to the second question above
- if there's no urgency, following the upstream's PR process to completion (else skip to next step).
- synching changes from the upstream PR process to the EWOrg PR
- finalizing the PR in EWOrg and get approval(s) for the PR
- once approved, working with the Maintainer to ensure your changes "run up the chain" of repos.
See picture which should help determine process.
If you notice a problem please open an Issue in the main model repository: create a new Issue in EWOrg/EW. We hope that a good Issue contains:
- A summary line describing the issue and context in brief
- A description that tries to answer the questions:
- What did you observe? What did you expect instead?
- What version(s) of EarthWorks (and/or externals) were you using? Were there any local modifications?
- What is the exact text of any error messages that seem important?
- Does this cause an issue on any supported system or within any supported compset?
This should trigger a discussion amongst the EarthWorks team of the problem which results in a decision of whether to address the issue (and hopefully how to address as well). A member of the EarthWorks team will then take on the Issue and will be responsible for working on changes and submitting them by the PR process (see Changes to submit).
Maintainer-specific.
Once a PR in an EarthWorks external has been accepted, there should be other related PRs in the "over-repos" that use the external. Those PRs contain some entry in Externals.cfg that has so far pointed to a branch in the developer's fork of the external. It's the maintainer's responsibility to:
- Ensure changes are merged to the correct branch in the external (and pushed to GH). Typically develop branch of the external.
- Ensure a tag is made (and pushed to GH) within the external with an appropriate tag name and body
- Find an "over-repo" PR that uses the changes to the external
- Edit the "over-repo" Externals.cfg to use the new external tag (from the EarthWorksOrg)
- Approve the "over-repo" PR and ensure these steps repeat until the same process is done for the main model repository.
See picture