Pip packaging at last!

[alexreinking]

This PR reworks our Python bindings and build system to support automatically packaging and deploying to (Test) PyPI.

Building LLVM here is a bear. Using caching has helped, but those caches are large and could expire. We should (later!) investigate ways to build suitable binaries using the buildbots. Building our custom manylinux docker containers on the buildbots is a start:

• https://github.com/halide/docker-images

This currently supports the following platforms:

• macOS 11+ arm64, x86-64 (no universal2 thanks to wheel size issues)
• Linux x86-64, glibc 2.28+ (in practice: Debian 10+, Ubuntu 18.04 LTS+, Fedora 29+)
• Windows x86-64

Wheels are provided for these platforms for each of Python 3.8-3.13.

Installing the halide pip package makes Halide available to both the Python interpreter and to CMake's find_package. It is therefore recommended to install into a virtual environment rather than to a systemwide interpreter (at least globally).

Depends on #8390

Fixes #4876
Fixes #5360

Other notable changes:

1. Our version number is now managed by tbump, which will keep the pyproject.toml, CMakeLists.txt, vcpkg.json, and HalideRuntime.h versions in sync.
2. Our python package version is determined by setuptools-scm, which uses git tags to determine the base version number, an incrementing .devNN tag, and a +gHASH local suffix computed from the git hash.
3. The above point requires us to tag commits changing the version number to the new number plus a .dev0 suffix. I have already done this for all historical commits (see for example v18.0.0.dev0)
4. Many of the Python packaging hacks have moved into the pyproject.toml-based build system, scikit-build-core.

Further bugs fixed:

1. Python is now modeled as an optional feature of the Halide CMake package. For backwards compatibility, it will continue to be loaded even if not specified and available. Now, it may be listed as in the REQUIRED COMPONENTS argument to find_package, which will raise an error if not present. In the future, the component will only be loaded if requested.
2. A bug in our detection of Apple ld support for linker scripts was fixed (some versions of Xcode error if a symbol is listed that doesn't exist)

Out of scope for this PR:

1. Splitting Halide into two Python packages: halide-bin for the Python-independent bits (i.e. libHalide) and halide for the Python bindings (depends on the former). This will improve CI performance and PyPI storage usage at the cost of considerable CI workflow complexity. The Python packaging ecosystem is not ready to build sets of dependent wheels at once.
2. Supporting arm64 Linux: Qemu emulation is too slow and the free GHA runners don't have native runners for this architecture. Using self-hosted runners might be a worthy alternative down the line.
3. Supporting 32-bit targets: This is largely untested anyway.
4. Switching to nanobind: (or the rumored pybind11 v3). Ideally we would be able to target the Stable or Limited ABI/APIs with our bindings. This would reduce the number of wheels we need to produce.

[alexreinking]

Something that might be principled / worth doing down the line is to Pip-package our LLVM binaries so they can be listed as a build-time dependency of our wheels. We could host them from the build master. Unfortunately, they are far too large to be hosted on PyPI. We might be able to further shrink the size by truncating un-needed files in bin/ (e.g. llvm-objdump), but we cannot simple exclude or delete them as LLVM's CMake package will complain about their absence.

Doing the same with wabt / flatbuffers might be profitable too. These are small enough to upload to PyPI, but I suspect that upstreams wouldn't appreciate us stealing those names. We'd have to coordinate with them or host from a private registry.

[alexreinking]

Something that took way longer to diagnose than it should have and that I'm saving here for posterity (and perhaps the search results of other unfortunate souls):

cibuildwheel allows you to set environment variables for Windows via the CIBW_ENVIRONMENT_WINDOWS environment variable. I had written in my YAML config:

env:
  CIBW_ENVIRONMENT_WINDOWS: >
    CMAKE_GENERATOR=Ninja
    CMAKE_PREFIX_PATH="${{ github.workspace }}/opt"

Woe on me for using double quotes! The string is parsed as if it were a bash command and ${{ github.workspace }} expands to D:\a\Halide\Halide so these backslashes become escape sequences! When using single quotes, the backslashes are literal.

I needed to SSH into the runner using mxschmitt/action-tmate@v3 to diagnose this.

[steven-johnson]

Supporting 32-bit targets: This is largely untested anyway.

IIRC we explicitly state that Python binding for Halide are only supported for 64-bit targets. IMHO it's not worth supporting 32-bit targets at this point in history.

[steven-johnson]

Absolutely brilliant.

[alexreinking]

Summoning the build bots for full review.

[alexreinking]

Failures are due to linux-worker-4 issues:

• missing Python package dependencies (c.f. Create Python virtual environment as part of build build_bot#129)
• missing native PNG libs
• missing -lz
• disconnections

Landing to confirm Test PyPI upload works. Docs to come.

[dragly]

Fantastic! Thanks for doing this 🎉 I am looking forward to be able to ‘pip install halide’ 😊

Did I understand correctly that the default wheels will support WebAssembly as the target? If so, that’s awesome for integrating Halide into Jupyter notebooks.

[alexreinking]

Fantastic! Thanks for doing this 🎉 I am looking forward to be able to ‘pip install halide’ 😊

Thank you for the kind words, @dragly! 😊

You can try it right now from the Test PyPI index:

$ pip install halide --pre --extra-index-url https://test.pypi.org/simple

When we release Halide 19.0.0 in a few weeks, then plain old pip install halide will work.

Did I understand correctly that the default wheels will support WebAssembly as the target? If so, that’s awesome for integrating Halide into Jupyter notebooks.

Yes. WebAssembly is now a required backend (along with X86). See #8344

I'm curious to learn more about the Jupyter notebook usage. Do you have an example of using Halide's WebAssembly backend in that context?

[steven-johnson]

When we release Halide 19.0.0 in a few weeks

We probably won't release a Halide 19 until LLVM 19 is final, I don't think there's a date for that yet, but it is in RC so hopefully not too long.

[alexreinking]

We probably won't release a Halide 19 until LLVM 19 is final, I don't think there's a date for that yet, but it is in RC so hopefully not too long.

19.1.0 final is expected on the 17th. Unless we have major blockers, our release shouldn't take long after that.

https://discourse.llvm.org/t/llvm-19-1-0-rc4-released/81039

[dragly]

You can try it right now from the Test PyPI index

Very nice! Worked right out of the box here.

I'm curious to learn more about the Jupyter notebook usage. Do you have an example of using Halide's WebAssembly backend in that context?

I have been playing around with a wrapper around IPython's display function for Halide objects. The goal is to make an interactive viewer where you can adjust the Param values using HTML sliders or other controls.

You can get pretty far on something like this without WebAssembly as well, but then you need a running Jupyter kernel.

With WebAssembly, you can include interactive viewers in an HTML version of a notebook and it will just keep working with static hosting. I have for instance been using Jupyter Book both for some personal projects and for internal documentation of our Halide code at work. So I am hoping to include some interactive visualizations there too.

The work-in-progress API looks something like this:

factor = hl.Param(hl.UInt(8), "factor", 0)
darken = hl.Func("darken")
darken[x, y, c] = image[x, y, c] / factor
slider = Slider(factor, min_value=1, max_value=10, step=1)
display_halide(darken, controls=[slider])

Which then results in output looking something like this:

In the implementation I am compiling the Halide Func to WebAssembly and then invoking Emscripten to link it and generate the final .js and .wasm files. These are then loaded by a <script> tag that I display together with some JavaScript glue code. And that code is rendered together with some HTML using IPython's display function.

The code is currently a mess. I first wanted to see if making such a viewer was at all doable. However, now that it has become so easy to include Halide in a Python project, I will probably write up a post about it and share it in the near future :)

[alexreinking]

@dragly -- This is super cool! Hope I get to play with it soon 🙂

[dragly]

@alexreinking I wrote up a post about it here: https://dragly.org/2024/09/08/interactive-halide/

I figured I might as well host it as a package now that Halide is coming to PyPI 🎉 The code is still very hacky, though. You should expect it to fail even with simple examples. But it should be usable with examples like the ones in the post.

And you need to have Emscripten installed and in PATH for it to work. If only there was a PyPI package for Emscripten too 😄