Treewide Nix reformat pass 1 [skip treewide]

[infinisil]

This is the next step towards accepted NixOS/rfcs#166 after #322512 and #326407, see also #322520 and NixOS/nixfmt#153 🎉.

After final improvements to the official formatter implementation, this commit now performs the first treewide reformat of Nix files using it. This is part of the implementation of NixOS/rfcs#166.

Only "inactive" files are reformatted, meaning only files that aren't being touched by any PR with activity in the past 2 months. This is to avoid conflicts for PRs that might soon be merged. Later we can do a full treewide reformat to get the rest, which should not cause as many conflicts.

A CI check has already been running for some time to ensure that new and already-formatted files are formatted, so the files being reformatted here should also stay formatted.

The @NixOS/nix-formatting will approve/merge this PR.

Verification

The reformatting commit is fully auto-generated by the code in https://github.com/infinisil/treewide-nixpkgs-reformat-script, see the commit message on how to do a complete verification.

Things done

• Merge and propagate latest nixfmt improvements and fixes
  • Update nixfmt: nixfmt-rfc-style: 2024-11-26 -> 2024-12-04 #361875
  • Update the CI-pinned Nixpkgs (master): ci: Update pinned Nixpkgs #363585
• Fully automate generation and verification of the formatting commit.
  • Check that nix-instantiate --parse doesn't change (add --arg check true to the verification build to do that, takes a bunch longer)
• Confirm that there are (almost) no rebuilds (See [Tracking issue] remove all src = ./. entries that unintentionally include their own .nix files #301014)
  • Can be done separately, non-blocking in Prevent some rebuilds for future Nix reformats #326430
• Make a Discourse post: https://discourse.nixos.org/t/nix-formatting-team-treewide-nixpkgs-formatting/56666
• Prepare the same PR for release and staging branches, so that they can be merged at the same time. This ensures that there aren't any problems when backporting PRs or merging staging
  • [release-24.05] Treewide Nix reformat pass 1 [skip treewide] #327796
    • Depends on [release-24.05] Update pinned nixpkgs version #364004
  • [release-24.11] Treewide Nix reformat pass 1 [skip treewide] #363995
  • [staging] Treewide Nix reformat pass 1 [skip treewide] #363999
  • [staging-next] Treewide Nix reformat pass 1 [skip treewide] #364000

---

This work is sponsored by Antithesis ✨

Add a 👍 reaction to pull requests you find important.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/formatting-team-meeting-2024-06-25/47661/1

[oxalica]

I'd say we should block on NixOS/nixfmt#153. There are also many issues about either awkward formatting result or implementation correctness. It's no good to race reformatting everything before that.

This was done with this script in ~40 seconds

This sounds too long to me. If the whole-repo checking takes too long, it blocks auto-checking in git commit hook.
fd -e nix --exec-batch nixfmt takes 1:49.27 (~110s) on my machine and nixfmt is single-threaded. nixfmt --check on formatted files takes almost the same time.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-57/47823/1

[infinisil]

@oxalica The reason I'm pushing for this is because there's a lot of churn from people reformatting individual parts on their own, @mweinelt can attest to that. Let's not let perfect be the enemy of good, the formatter works and is unlikely to change drastically. We should definitely block on correctness bugs like NixOS/nixfmt#197 though, this is being worked on :)

Regarding the time it takes for a full format, the solution I'd propose is to use a pre-commit hook that only formats files that were changed. This also nicely aligns with CI only requiring changed files to be formatted. I'd be okay with blocking on that for this PR, I'll add it to the TODO items.

In addition to that, if CI fails, it shows the nixfmt command needed to make CI pass, only listing the files that need reformatting, see here.

Alternative: treefmt

One could also imagine using treefmt because it caches results, but this will be slightly problematic when we change the formatter in the future and need to do the same multi-pass approach to avoid merge conflicts. There's going to be times when not the entire codebase is formatted in the same way, and treefmt would mess that up.

[oxalica]

use a pre-commit hook that only formats files that were changed

That's kinda non-trivial to me, and you need to retrieve file list from git log on an unknown number of commits. Would be good to have an example (maybe in README) for copy-paste.

Also I opened NixOS/nixfmt#211 for tracking this.

[infinisil]

That's kinda non-trivial to me, and you need to retrieve file list from git log on an unknown number of commits. Would be good to have an example (maybe in README) for copy-paste.

Since it's the pre-commit hook, we only have to worry about the single commit that is being made, so I don't think it will be bad, but I'll see it when I try 😆. And we can use a shellHook automatically pre-commit hooks upon entering the shell, e.g. https://github.com/cachix/git-hooks.nix also works in that way.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/brainstorming-for-rfc-assimilate-home-manager-into-nixpkgs-monorepo/48230/13

[Aleksanaa]

I'd say we should block on NixOS/nixfmt#153. There are also many issues about either awkward formatting result or implementation correctness. It's no good to race reformatting everything before that.

The good thing here is we can skip formatting if something weird really happens, because it doesn't make subsequent CI check fail.

[emilazy]

I’d like to recommend a pre-push hook instead of a pre-commit one. It would still check all the commits that are to be pushed. The advantage is that it is easier to slice and dice existing commits and make imperfect work‐in‐progress commits to clean up later without having the formatting step get in the way. It also prevents making committing any slower, whereas pushes are already slow. For me, the experience of using a pre-commit hook is like having to satisfy a pedantic spell‐checker while trying to get into a flow state writing a stream‐of‐consciousness draft.

Another advantage is that tools like git-revise cannot necessarily run hooks due to their in‐memory design, which wouldn’t affect a pre-push hook; it’s more reliable to do the checks at the actual publication bottleneck.

If you need any help writing an efficient and correct Git hook for this, please ping me.

[infinisil]

To make this PR easier to review, I split off the enforcement for new files into #326407. So this PR is only for the treewide format now! Also note:

• I added instructions to the PR description on how the entire contents of this PR can be reproduced, allowing anybody to easily check it
• Some nixfmt fixes should be merged and propagated before this is merged, see the PR description

[infinisil]

I walked through the methodology of this work with ~12 people in the merge party call, no complaints, merging! 🎉

[infinisil]

Turns out the "reformat everything on master/staging/staging-next together" approach didn't quite work out: master couldn't be automatically merged into staging-next anymore (same for staging-next into staging). Thanks to @wegank for taking care of fixing that by manually doing the staging merge!

[mightyiam]

Thanks to the works of inspired, motivated and provided individuals, we can has nice things! Thanks, everyone involved!

[trofi]

I have about 350 local patches against nixpkgs. Most of them are conflicting now. Is there a standard way to automatically rebase most of them with formatting applied? Or I'll have to hack up something locally?

[arianvp]

There's this which might help: #363759

[Pandapip1]

I have about 350 local patches against nixpkgs. Most of them are conflicting now. Is there a standard way to automatically rebase most of them with formatting applied? Or I'll have to hack up something locally?

If these are generally applicable patch sets, can I suggest that they be sent to nixpkgs? This could have been avoided that way:

Only "inactive" files are reformatted, meaning only files that aren't being touched by any PR with activity in the past 2 months. This is to avoid conflicts for PRs that might soon be merged. Later we can do a full treewide reformat to get the rest, which should not cause as many conflicts.

[trofi]

Is there a standard way to automatically rebase most of them with formatting applied?

Ended up doing the following:

# 1. Create empty commit:
git commit --allow-empty -m "EMPTY commit: will absorb relevant formatting changes"

# 2. Move the empty commit to queue beginning:
git rebase -i --keep-base
# Move "EMPTY commit: will absorb relevant formatting changes" entry from last line to the first line.

# 3. Get affected formatted files. Pick the commit from `.git-blame-ignore-revs` for the relevant branch. For `staging` it is `667d42c00d566e091e6b9a19b365099315d0e611`
FORMATTED_FILES=$(git diff --name-only 667d42c00d566e091e6b9a19b365099315d0e611^..667d42c00d566e091e6b9a19b365099315d0e611 -- $(git diff --name-only origin/staging...staging) | tr $'\n' ' ')`

# 4. Reformat:
FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --tree-filter "nixfmt $FORMATTED_FILES" -- $(git merge-base origin/staging staging)..
   ...
   Rewrite 6fc0a951e9b7a7e3f80628ca0a6c4c9f54fd2dd6 (56/327) (65 seconds passed, remaining 314 predicted)
   ...
   Rewrite c20df82da66da6521f355af508bfedc047cffa64 (326/326) (1183 seconds passed, remaining 0 predicted)
   Ref 'refs/heads/staging' was rewritten

# 5. Rebase past the reformat (as one would normally do the update):
git rebase -i

# Done