Speeding up performance

[vvuk]

I've got a very large repository, and many branchless operations take a long time (eg 60s+ for processing checkout / examining local history / walking commits on a M3 MacBook Pro). What can I do to speed this up? As it is I can't really use branchless effectively in this repo, because even a simple git move to move one commit onto a new remote head takes multiple minutes.

Is there some way to flag "I don't care about commits earlier than this date/this commit", or similar? git hide doesn't work on ancestors, and I'm not sure if it would speed things up (instead of just hiding).

[claytonrcarter]

very large repository

Can you offer any more context on this? 100k? 1m? More? I have no experience with very large repos, but one of the design goals of the project is to scale to large repos, and I recall seeing that someone is using it effectively with on chromium, which has 1.3 million commits.

The one place where I have seen performance degrade quickly, though, is in repos with lots of branches. (I've seen git next churn away for 10+ minutes in repos with 13k branches.) In particular, the smartlog is slow and noisy if you have a lot of branches. One thing to try is changing your default revset. For example, you may try setting that to stack() | @ so only see your current stack of commits.

Is there some way to flag "I don't care about commits earlier than this date/this commit", or similar?

You can look at the revset docs to see what's available, but you could try something like draft() & committer.date('after:2 weeks ago') when working with revsets.

[vvuk]

git rev-list --count -all says 1,260,064. We definitely have lots of branches though... 7701 under remotes/origin, so double that counting the local copies; I need to fix that though since I definitely don't need to clone all those locally at all. The history is also very very merge-heavy. This was a former mercurial repository that was converted to hg; the smartlog graph is comical, I often get hundreds of | parallel branch markers on commits.

I'll try changing the default revset; that may actually help because the thing that was taking all the time on some operations may have been the smartlog at the end. Your suggestion of stack() | @ completes in ~1-2s, the original default revset is 2+ minutes.

[claytonrcarter]

7701 under remotes/origin, so double that counting the local copies; I need to fix that though since I definitely don't need to clone all those locally at all

I may need to be fact checked on this, but I believe that branchless completely ignores remote branches. I can certainly believe that ~8k local branches would contribute to the issues you're seeing, though.

very very merge-heavy

This shouldn't be a big deal. The smartlog can render merges (though it's not super intuitive output and I recall that there may be some outstanding issues with it).

completes in ~1-2s, the original default revset is 2+ minutes

Great!

[vvuk]

Hmm. After a git fetch of two specific branches (so not a ton of branch changes), git triggered a git gc. After this, I now have git-branchless hook reference-transaction committed running for 30+ minutes at 100% cpu. Any ideas?

[arxanas]

• git-branchless operations scale linearly in the number of references, so you want to substantially reduce your reference count to just those that you're working on.
  • It has to scan every reference on startup to build/update its internal DAG structure, so that commits/revsets referenced as part of the command are known to git-branchless and evaluate to the correct values.
  • Most likely, you'll want to delete the majority of your existing references and modify your default fetch refspec to only fetch branches that you're working on.
• Merge-heavy history should be more or less fine for performance, as per the reported measurements for the segmented changelog's performance from Sapling SCM, which we use.
• The actual performance of git branchless smartlog is not good when there are a lot of nodes (possibly quadratic for no particularly good reason?; could be fixed).
• Unfortunately, the reference-transaction hook issue doesn't have a good resolution. It might be best to uninstall git-branchless, perform the GC, then reinstall it.
  • Git itself is not grouping the reference updates into a single transaction, which means we incur the hook overhead for each individual reference. This might improve as reftable is developed/integrated into GIt.