formulae_dependents: apply some optimisations #841
Conversation
1. Doing `brew install --only-dependencies` and `brew postinstall` can be quite expensive. Let's defer this to when we're sure we have dependents to test. 2. Calling `brew uses` is slow, because this requires traversing the dependency tree. [*] Let's avoid doing this unless we know we really need to. Here, we test for "needing to" by checking if another `.rb` file in the tap that might be a formula contains a `depends_on` line declaring a dependency on the formula being tested. 3. Restrict the second `brew uses` call to when we are building dependents from source, since that is the only instance where we are interested in the build dependents. While we're here, make sure to call `brew postinstall` on all dependencies that were rebuilt, and not just the one being tested currently. This may address Homebrew#805. Locally, this results in the following speed up for a formula with no dependents: Before ( brew test-bot --only-formulae-dependents --testing-formulae=hello --dry-run) 28.95s user 6.70s system 76% cpu 46.875 total After ( brew test-bot --only-formulae-dependents --testing-formulae=hello --dry-run) 0.91s user 1.02s system 51% cpu 3.738 total This makes testing formulae with dependents slightly slower. However, the vast majority of formulae in Homebrew/core have no dependents (on macOS, at least), so this is likely a net win for the average workflow. [*] Potential future optimisation: calling `Dependency.expand` directly might give us better opportunities to exploit caching.
| next false unless child.file? | ||
| next false unless child.extname == ".rb" | ||
|
|
||
| child.read.include? "depends_on \"#{formula_name}\"" |
There was a problem hiding this comment.
I'd really love to avoid something this hacky as this when we have RuboCop AST methods to do this sort of thing.
There was a problem hiding this comment.
Can we construct the AST without loading the formula? Because that's part of what's slow here.
There was a problem hiding this comment.
For example, doing this instead:
has_dependents = formula.tap.formula_names.any? do |name|
Formula[name].deps.map(&:name).include? formula_name
endWe get:
( brew test-bot --only-formulae-dependents --testing-formulae=hello --dry-run) 12.84s user 2.96s system 68% cpu 23.128 total
It's faster than just doing brew uses, but also a about 6 times slower than just matching text.
| Utils.safe_popen_read("brew", "uses", *uses_args, "--include-build", formula_name) | ||
| .split("\n") | ||
| # We care about build dependents only if we are building them from source. | ||
| if args.build_dependents_from_source? |
| # Install formula dependencies. These will have been uninstalled after building. | ||
| test "brew", "install", "--only-dependencies", formula_name, | ||
| env: { "HOMEBREW_DEVELOPER" => nil } | ||
| return if steps.last.failed? | ||
|
|
||
| # Restore etc/var files that may have been nuked in the build stage. | ||
| test "brew", "postinstall", formula_name | ||
| formula_dependencies = Utils.safe_popen_read("brew", "deps", formula_name).split("\n") | ||
| # Some dependencies will need postinstalling too. |
There was a problem hiding this comment.
It's not clear to me why this is the case, can you explain more?
There was a problem hiding this comment.
Suppose testing formula includes foo and bar, and both have a common dependent baz.
Whenever we test foo first, we do brew postinstall foo but we skip it for bar, but that's wrong. The reasons why we need to brew postinstall foo apply to needing to brew postinstall bar too.
| # Install formula dependencies. These will have been uninstalled after building. | ||
| test "brew", "install", "--only-dependencies", formula_name, | ||
| env: { "HOMEBREW_DEVELOPER" => nil } | ||
| return if steps.last.failed? | ||
|
|
||
| # Restore etc/var files that may have been nuked in the build stage. | ||
| test "brew", "postinstall", formula_name | ||
| formula_dependencies = Utils.safe_popen_read("brew", "deps", formula_name).split("\n") |
There was a problem hiding this comment.
Are we calling brew deps anywhere else? If so, worth caching?
There was a problem hiding this comment.
I'll have a look, but I don't think so it's used elsewhere.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
brew install --only-dependenciesandbrew postinstallcanbe quite expensive. Let's defer this to when we're sure we have
dependents to test.
brew usesis slow, because this requires traversing thedependency tree. [*] Let's avoid doing this unless we know we really
need to. Here, we test for "needing to" by checking if another
.rbfile in the tap that might be a formula contains a
depends_onlinedeclaring a dependency on the formula being tested.
brew usescall to when we are buildingdependents from source, since that is the only instance where we are
interested in the build dependents.
While we're here, make sure to call
brew postinstallon alldependencies that were rebuilt, and not just the one being tested
currently. This may address #805.
Locally, this results in the following speed up for a formula with no
dependents:
Before
After
This makes testing formulae with dependents slightly slower. However,
the vast majority of formulae in Homebrew/core have no dependents (on
macOS, at least), so this is likely a net win for the average workflow.
[*] Potential future optimisation: calling
Dependency.expanddirectlymight give us better opportunities to exploit caching.