Hops: experimental Homebrew bottle installer and relocator

[justenstall]

Hello! I have been working on an experimental Homebrew bottle installer called Hops: https://github.com/act3-ai/hops

The goal was to use Homebrew in air gapped networks, which is enabled by a command that copies bottles and their metadata to a standalone OCI registry and an install command that uses the standalone registry. The project was written in Go, so I was able to use concurrency to improve performance, especially for downloads.

Would love to share more about the experiment if there is any interest
@MikeMcQuaid

[MikeMcQuaid]

@justenstall This sounds interesting, thanks for sharing. Could you provide more details on how this works for the hops runner for the end-user? What's your personal use-case here? Thanks!

[justenstall]

Our current scenario is:

• We have an air gapped network (no outside internet connection)
• We need to use various tools within the network
• We produce a .deb package containing all of the necessary tools
• The .deb package is uploaded to a package registry in the air gapped network so developers can install the tools there

We would like to use Homebrew for this because:

• There are Homebrew Formulae for almost all of the tools we are using (but there are not always existing Debian packages)
• Homebrew is cross-platform for macOS and Linux (current approach only works for Linux)
• We already use Homebrew to have our developers install internal tools from a private registry (with a custom DownloadStrategy): act3-ai/tap

Hops lets us use Homebrew bottles in the air gapped network with an OCI registry as the only requirement.

With Hops:

1. The administrator of the air gapped network runs hops copy to copy all necessary bottles and their metadata to a destination OCI registry
   • The command takes a list of formulae to copy
2. The contents of that OCI registry are uploaded to the air gapped network's OCI registry
3. End users (in the air gapped network) configure their Hops CLI to use the OCI registry as the single source of truth for bottles/metadata
   • Set by --registry command line flag, HOPS_REGISTRY environment variable, or .registry.prefix in Hops' YAML config file
   • Works similar to the behavior of HOMEBREW_BOTTLE_DOMAIN, HOMEBREW_ARTIFACT_DOMAIN, and HOMEBREW_API_DOMAIN but expects all content to be available in the registry
4. When an end user runs hops install ___, the formula is installed as normal, but the API data and bottle file are downloaded from the same registry

That's a lot of information, let me know if there's anything I can clarify.

[MikeMcQuaid]

Thanks @justenstall! A few more questions.

2. The contents of that OCI registry are uploaded to the air gapped network's OCI registry

How does this occur? Is hops used here or other standard Unix tools?

4. When an end user runs hops install ___, the formula is installed as normal, but the API data and bottle file are downloaded from the same registry

Does hops install basically do "brew fetch but from HOPS_REGISTRY" and then "brew install as normal from Homebrew but relying on the cached files?

• Works similar to the behavior of HOMEBREW_BOTTLE_DOMAIN, HOMEBREW_ARTIFACT_DOMAIN, and HOMEBREW_API_DOMAIN but expects all content to be available in the registry

Out of interest: what is it that made you decide to create a new tool rather than use these variables? Another way of asking, perhaps: what about these variables was too limited for your use-case?

[justenstall]

2. The contents of that OCI registry are uploaded to the air gapped network's OCI registry

How does this occur? Is hops used here or other standard Unix tools?

Other tools are used for this.

4. When an end user runs hops install ___, the formula is installed as normal, but the API data and bottle file are downloaded from the same registry

Does hops install basically do "brew fetch but from HOPS_REGISTRY" and then "brew install as normal from Homebrew but relying on the cached files?

Not quite. brew fetch and brew install rely on the formulae.brew.sh API (or the homebrew-core Git repository if the API is disabled). hops install (when using HOPS_REGISTRY) does not rely on the formulae.brew.sh API, because the API data is uploaded to the registry.

• Works similar to the behavior of HOMEBREW_BOTTLE_DOMAIN, HOMEBREW_ARTIFACT_DOMAIN, and HOMEBREW_API_DOMAIN but expects all content to be available in the registry

Out of interest: what is it that made you decide to create a new tool rather than use these variables? Another way of asking, perhaps: what about these variables was too limited for your use-case?

To use the existing variables, we would have had to mirror two data sources:

1. Either the formulae.brew.sh API or the homebrew-core Git repository
2. The contents of the ghcr.io/homebrew/core container registry

That could have worked, but it was theoretically possible the mirrored data sources could get out of sync (also we did not want to host a mirror of the API), so we came up with a way to bundle the API data for each formula with the bottle files in the container registry so the bottles are self describing. The rest of hops was only necessary because brew doesn't know how to consume the self describing bottles.

[MikeMcQuaid]

Other tools are used for this.

Are you willing/able to detail these tools or is this your "secret sauce"? No worries if so, just interested in figuring out how the whole workflow goes here.

hops install (when using HOPS_REGISTRY) does not rely on the formulae.brew.sh API, because the API data is uploaded to the registry.

So is Homebrew not doing e.g. relocation of files after installation and the usual preinstall/postinstall checks provided by brew install?

so we came up with a way to bundle the API data for each formula with the bottle files in the container registry so the bottles are self describing.

Nice. This is something we'd consider upstreaming, if you were interested in either working on this or detailing the approach so someone else could?

[justenstall]

Are you willing/able to detail these tools or is this your "secret sauce"? No worries if so, just interested in figuring out how the whole workflow goes here.

We use an internal tool for mirroring OCI images/artifacts. It is maintained on our private Git host but there is an open source mirror on GitLab.com: https://gitlab.com/act3-ai/asce/data/tool

So is Homebrew not doing e.g. relocation of files after installation and the usual preinstall/postinstall checks provided by brew install?

No, Hops does not depend on Homebrew. I have reimplemented the fetch, pour, link, etc. functionality in hops install.

Nice. This is something we'd consider upstreaming, if you were interested in either working on this or detailing the approach so someone else could?

Absolutely! Let me know how you'd like to start that process. I can open an issue, but meeting to discuss the details beforehand might be helpful.