New release notes process: notes in every PR

[huonw]

I've now merged #20850, which adds a new CI check: release notes are required.

This is a new process that will have us maintain release notes in markdown files in docs/notes as we go, so that it's easier:

• for users to see a human-curated summary of what's new in the development branches,
• for Pants maintainers to do stable releases (and thus we'll be able to do them more often!).

Here's my current thinking of the process:

• for contributions (i.e. what do when opening a PR)
• for reviewing a PR
• for doing a release
• plus some notes about this transition period

Please ask if anything is unclear. Process change is hard!

(NB. there's some discussion in #20850 about next steps, including reference to #19247.)

For contributions

1. Open a PR as usual
2. You may get a reminder to add release notes.
   • For some changes, no release notes are required (see below), a Contributor/Maintainer will just need to label the PR appropriately.
   • For others, they should be added: find the right file in docs/notes/... (usually, the biggest number, 2.22.x.md currently) and add a sentence or a paragraph.
3. If you happen remember in advance, feel free to add release notes before opening the PR.

Non-exhaustive examples of changes that don't need release notes:

• minor doc improvements
• fixes to features that haven't been released yet (although sometimes the release notes for that feature can be updated!)
• internal CI changes (covered by being labelled with category:internal usually)

Use your discretion to decide if something isn't worth calling out, and feel free to ask on the PR or in slack.

For reviewing a PR

When reviewing a PR, it may have releases notes already, in which case:

• review them like any documentation
• check they're in the right file: the first 2.x.0 stable release that they'll be released in. For example:
  • if a PR is just merging to the main branch, and it's is leading up to the 2.22 releases (as it is now), release notes should be in docs/notes/2.22.x
  • if a PR will be cherry picked back to 2.21.x (2.21.0 is not yet released), they can go in docs/notes/2.21.x.md

If it doesn't have release notes, the release notes CI job will fail (unless labelled), and will remind us to either:

• add release notes
• add a label:
  • release-notes:not-required to opt-out of release notes when not required (see above for a hint)
  • PRs labelled category:internal don't require release notes (but can have them, if required)

For doing a release

When doing a dev or RC release, nothing has changed.

There are changes for other release:

• a0: when we do a release like 2.22.0a0, the main branch goes from being for 2.22 to being for 2.23, so release notes need to switch to a new file (by default). Require release notes or explicit opt-out in every PR #20850 updates the release procedure with what we should be doing here.
• stable: release notes are already written, but it's probably worth doing a once-over to check they make sense!

For this transition

With the transition to a new process, they'll be a bunch of existing PRs that don't have release notes and may not have the new CI check run on them before merging. This means we may have some cool new feature that merges but isn't included in release notes... so no-one knows about it!

I (@huonw) have gone through the most likely candidates (PRs approved now) and asked for an update, and I'll keep an eye on what's merged and back-fill the notes if we miss something.

[thejcannon]

if a PR will be cherry picked back to 2.21.x (2.21.0 is not yet released), they can go in docs/notes/2.21.x.md

This could be enforced (via the cherry pick label).

I'm guessing CI will fail on the cherry pick PR if you didn't touch the correct release notes for the cherry-picked version?

---

Does this mean cherry pick PRs will also be picking a future-releases notes on older branches? I assume that's ok/expected?

[huonw]

I'm guessing CI will fail on the cherry pick PR if you didn't touch the correct release notes for the cherry-picked version?

Three layers to this:

1. With the current branches, no, since we haven't cherry picked the CI enforcement back
2. In future, no too, because the CI check just looks for any change to docs/notes, not a change to a particular version
3. And even if we fixed that, almost certainly no, also: if a PR adds to the latest release notes (2.22.x.md) and is cherry picked back to 2.21.x, that file won't existing on that branch, so there'll be a merge conflict instead. (I think this is your second question!)

I haven't progressed in my thinking beyond this, #20850 "Problems":

3. this'll make cherry-picking more awkward and fail more often, e.g. if a change has release note in 2.22.x.md and we cherry pick it to the 2.21.x branch, that file won't exist and the cherry picker will fail. I think this is probably tolerable for now (at least, worth trying). There's a potential few mitigations (I think the first is the better use of energy):
   1. switching to a different approach to notes where they end up in separate dedicated files (see "Future work")
   2. making the cherry-picker script more intelligent, e.g. ignore notes or copy them across magically

The "future work" that helps with this would be the towncrier (or similar) approach of separate files for each PR that get aggregated when creating release notes. Those files can be cherrypicked around just fine, and do the right thing more smoothly (e.g. if they get picked to the 2.20.x branch now, they could be used to create the release notes for 2.20.2; magic).

(I've already spent a lot of my recent Pants volunteering on "process" stuff, not "code" stuff, so won't be pushing that piece of future work. Maybe someone else is enthused and can start on it.)

[thejcannon]

making the cherry-picker script more intelligent

That makes sense. "Only copy the notes whose version matches the target branch"

[tdyas]

#21015 is a "make the cherry-picker script more intelligent" PR.

[huonw]

I'll keep an eye on what's merged and back-fill the notes if we miss something.

Nothing missed so far.

I have been checking with this script (for all the commits to main after 2505e54, find the ones that don't mention docs/notes/ at all):

for rev in $(git rev-list upstream/main ^2505e54f2d); do
    if ! git show --name-only $rev | grep '^docs/notes/' > /dev/null; then
        git show --name-only $rev
        echo
        echo
    fi
done