Isolate macOS wheel builds from Homebrew #8497

freakboy3742 · 2024-10-25T01:55:57Z

The existing macOS cibuildwheel configuration relies on Homebrew to provide some dependencies. This clearly works in practice, but there are some issues associated with this choice.

Firstly, it is destructive when used on a local build machine. The wheel dependency script invokes brew remove --ignore-dependencies (which can leave the host machine in a broken, and potentially difficult to restore state); and requires the use of sudo to make changes in the /usr/local tree. This means it is difficult to test cibuildwheel changes locally, or to recommend using cibuildwheel as a local build solution.

Secondly, Homebrew builds could be incompatible with Pillow builds. The MACOSX_DEPLOYMENT_TARGET for Homebrew dependencies is fixed by the Homebrew build chain, rather than the Pillow build tools, so a change in Pillow's configuration may not be satisfied by Homebrew's build configuration. This isn't an issue at present, but it could easily become one in future.

Related to this, there is also the potential that a change in a Homebrew recipe could lead to the inadvertent introduction of additional binary libraries into the delocated macOS binaries. This is especially problematic with the fribidi, raqm and freetype dependedencies which are included as "vendor" sources.

However, the real motivation for proposing this change is that it is a precursor for PEP 730-compliant iOS builds. If Homebrew is on the build path when compiling iOS binaries, compiler tooling will often find an ARM64 Homebrew binary and attempt link it into an iOS library - which then (predictably) breaks. So - Homebrew isolation is essential for iOS compilation. Since the build processes for iOS and macOS are very similar, and Homebrew isolation was necessary for iOS, adding Homebrew isolation for macOS will simplify any future iOS patch. It also provides a way to submit iOS support in a series of smaller pieces, rather than a single "monster" patch. Plus, it provides all the side benefits of isolation described above.

An overview of the changes:

Uses the build folder as a location for all dependency builds. This prevents the root checkout from becoming dirtied by dependency artefacts, simplifying cleanup, and avoiding the need for extensive modifications to the .gitignore file.
Sets up an isolated build prefix in the build folder (./build/deps) where dependencies can be installed, and make installs all dependencies into that path.
Forces PATH to be a "clean" environment that only includes bare system tools, plus the Python binary being used for the build, and the isolated build prefix. macOS doesn't include most of it's development libraries in /usr or /usr/local, instead using a path provided by the macOS SDK (which is configured as part of the compiler toolchain).
Adds a build of pkg-config to the dependencies. This is the one build tool that autotools and cmake often require that Xcode doesn't provide. Providing a custom build of pkg-config also ensures that no dependencies other than the ones provided by cibuildwheel are included. The build recipe that is used is derived from the Homebrew recipe.
Adds a build of libXdmcp, to avoid a dependency on the Homebrew-provided version.
Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.
Updates the pinned multibuild version to the current devel branch hash. This is to incorporate Python3.13 support, plus a number of fixes required to support installs into locations other than /usr/local, and corrections to LIBDIR handling for platforms that use the lib64 suffix.
Adds a dependencies-prefix configuration option to setup.py. This allows the user to provide a specific location to look for dependencies, rather than the series of Fink/MacPorts/Homebrew fallbacks that currently exist.

.github/workflows/wheels-dependencies.sh

.github/workflows/wheels-test.sh

radarhere · 2024-10-25T02:13:13Z

.github/workflows/wheels-dependencies.sh

@@ -23,6 +23,8 @@ OPENJPEG_VERSION=2.5.2
 XZ_VERSION=5.6.3
 TIFF_VERSION=4.6.0
 LCMS2_VERSION=2.16
+RAQM_VERSION=0.7.1


Suggested change

RAQM_VERSION=0.7.1

RAQM_VERSION=0.10.2

It definitely makes sense to use the most recent available version; but pillow-depends-main doesn't include this version. What's the procedure for updating that repo?

(Related: pkg-config, libXdmcp, and an updated webp sources should probably be added to that project)

The theory is that pillow-depends-main is just a more reliable source for the dependencies, and PRs should work without it.

Once this is merged, I will likely add them there.

freakboy3742 · 2024-10-25T02:41:02Z

The manylinux failures look like they'll be fixed by this PR to multi build.

The x86_64 macOS failures are probably a leakage from /usr/local that I haven't accounted for. I'm looking into that one now.

I'm not sure how to interpret the coverage drop, though...

freakboy3742 · 2024-10-25T05:53:53Z

@radarhere I've updated the pin for multibuild to include the (just merged) PR reverting the libjpeg-turbo CMAKE_INSTALL_LIBDIR change; and I've addressed some additional locations where Homebrew can leak in on x86_64.

nulano · 2024-10-25T09:32:17Z

Related to this, there is also the potential that a change in a Homebrew recipe could lead to the inadvertent introduction of additional binary libraries into the delocated macOS binaries.

This has already happened many times in the past, so thanks for working on this.

Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.

I'm not sure if I understand what you mean here. We need to use a vendored version of raqm in wheels for license reasons. When using raqm=vendor, we build our own version of raqm from https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/raqm. Similarly, using fribidi=vendor does not actually build fribidi, but instead custom code that can load fribidi at runtime, see https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/fribidi-shim. The Homebrew version of fribidi is only used for testing the built wheels.

freakboy3742 · 2024-10-28T01:23:18Z

Adds builds of fribidi and raqm, rather than using the Homebrew versions as "vendored" versions. The setup.py configuration passed in by cibuildwheel has been modified from the default on other platforms to not use the vendor version.

I'm not sure if I understand what you mean here. We need to use a vendored version of raqm in wheels for license reasons. When using raqm=vendor, we build our own version of raqm from https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/raqm. Similarly, using fribidi=vendor does not actually build fribidi, but instead custom code that can load fribidi at runtime, see https://github.com/python-pillow/Pillow/tree/main/src/thirdparty/fribidi-shim. The Homebrew version of fribidi is only used for testing the built wheels.

Thanks - that's some helpful context. I was seeing the missing fribidi detection during the build and interpreting that as something that was part of the move away from homebrew; but I see now that Pillow is loading from Homebrew by design in this instance. That also might explain the coverage drop (as supplying fribidi means any runtime loading code won't be in use). I'll take another swing at that part (and resolve the linux build issues as well).

Out of interest - what's the license issue with raqm? The source code indicates It looks to be MIT licensed... did it used to be GPL (or something else problematic?)

hugovk · 2024-10-28T05:07:36Z

Out of interest - what's the license issue with raqm? The source code indicates It looks to be MIT licensed... did it used to be GPL (or something else problematic?)

From #2753 in 2017:

We can't currently distribute a binary of Pillow with support for raqm due to license incompatibility, because while libraqm is MIT licensed, it links to GPL libraries.

FriBiDi is LGPL.

Since then, Raqm added support for Apache-licensed SheenBiDi in 2021: HOST-Oman/libraqm#138, HOST-Oman/libraqm#139.

Perhaps we could switch.

nulano · 2024-11-04T19:04:23Z

The shim fribidi builds is used in the wheel, with Homebrew providing the fribidi binary on macOS in the test environment

Thanks, that looks better.

There still seems to be something wrong with freetype on arm64, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:14308

  SKIPPED [2] Tests/test_imagefont.py:1081: FreeType compiled without brotli or WOFF2 support

But it was built with brotli, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:7768:

    brotli:        yes (pkg-config)

We need to support this for #6554.

We could also consider compiling libwebp before libtiff so that WEBP-compressed tiff files can be read (AFAIK webp only looks for libtiff so that it can compile companion tools, but does not actually use it in the library - as I mentioned in #6562). But we don't need to do it now since the test was already being skipped on macOS and Linux before this PR.
https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:15489:

  SKIPPED [1] Tests/test_file_libtiff.py:905: WEBP compression support is not configured

freakboy3742 · 2024-11-04T22:04:13Z

There still seems to be something wrong with freetype on arm64, https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:14308

Weird. My immediate guess is that it might have something to do with the fact that a vendored version of fribidi is being used, which might not be triggering brotli support (or at least not triggering the test). I'll take a look and see what is going on.

We could also consider compiling libwebp before libtiff so that WEBP-compressed tiff files can be read (AFAIK webp only looks for libtiff so that it can compile companion tools, but does not actually use it in the library - as I mentioned in #6562). But we don't need to do it now since the test was already being skipped on macOS and Linux before this PR. https://github.com/python-pillow/Pillow/actions/runs/11604707482/job/32314276332?pr=8497#step:5:15489:

I saw that, and ultimately gave up trying to fix it - firstly because I was trying to limit the scope of the changes, but also because I'm not sure it can be fixed. libwebp has a dependency on tiff... but then lcms2 needs tiff; and openjpeg needs libpng, tiff, and lcms2. So - I'm not sure this can be fixed (or, at least, it can't be fixed without some sort of 2-pass compile and the use of dynamic linking, which iOS can't use anyway).

nulano · 2024-11-04T22:11:52Z

Weird. My immediate guess is that it might have something to do with the fact that a vendored version of fribidi is being used, which might not be triggering brotli support (or at least not triggering the test). I'll take a look and see what is going on.

The test skip is triggered by a specific error raised by the freetype library which itself does not use fribidi.

libwebp has a dependency on tiff... but then lcms2 needs tiff; and openjpeg needs libpng, tiff, and lcms2

AFAIK all of these dependencies are just for the binary utilities that we don't need so we can disable them with relevant compile flags; the libraries themselves do not have dependencies between them (except for libtiff optionally using libwebp). But I agree that it might be best to leave that for a separate PR.

freakboy3742 · 2024-11-06T06:28:48Z

@nulano I've found the culprit - Homebrew's copy of freetype was leaking into the test environment.

By using DYLD_LIBRARY_PATH pointed at Homebrew's lib foldler, all of Homebrew's libraries become available, and take take precedence over the ones provided by the wheel. As a result, Homebrew's copy of freetype is used, which doesn't include Brotli support. I presume this wasn't an issue previously because an explicit DYLD_LIBRARY_PATH comes earlier in library resolution precedence; when Homebrew was being used implicitly, Homebrew's library resolve after Pillow's bundled dependencies.

This can be fixed by pointing the DYLD_LIBRARY_PATH at the cellar version of the vendored fribidi library (which is the only library we actually need from Homebrew). I've pushed that update (update: I'm looking at the linting error now...)

The update includes 2 other small changes:

The location of the "homebrew isolated" macOS build is slightly altered - the extra darwin folder became necessary when I started working on the iOS build, and the build folders for macOS started colliding with iOS.
I've added some safety checks against some uses that are valid cibuildwheel invocations, but will break if you try to use them. Specifically, CIBW_ARCHS="arm64 x86_64", and not specifying CIBW_ARCHS at all, are both valid cibuildwheel setup, but Pillow doesn't allow it because the before_all step requires a single explicit architecture. This bit me a couple of times before I worked out what was happening; I figured it couldn't hurt to raise an explicit error when we know it won't work.

.github/workflows/wheels-dependencies.sh

nulano · 2024-11-07T19:32:15Z

Thanks, looks good to me now.

Co-authored-by: Andrew Murray <[email protected]>

.github/workflows/wheels-dependencies.sh

Co-authored-by: Andrew Murray <[email protected]>

radarhere · 2024-11-12T11:32:29Z

setup.py

@@ -448,7 +454,7 @@ def _remove_extension(self, name: str) -> None:
    def get_macos_sdk_path(self) -> str | None:
        try:
            sdk_path = (
-                subprocess.check_output(["xcrun", "--show-sdk-path"])
+                subprocess.check_output(["xcrun", "--show-sdk-path", "--sdk", "macosx"])


What was the reason for this change?

Explicitness. Xcode can manage multiple SDKs (iOS being an obvious one); adding the explicit --sdk macosx SDK ensures that you're definitely getting the macOS one. That will be the default on almost every install of Xcode, but if you have a stray SDKROOT environment variable from some other activity, it could be inadvertently pointing at an iOS, tvOS, visionOS or MacCatalyst SDK.

.github/workflows/wheels-test.sh

Disable platform guessing instead of adding dependencies-prefix

.github/workflows/wheels-dependencies.sh

radarhere · 2024-11-16T05:03:22Z

.github/workflows/wheels-dependencies.sh

+    # This essentially duplicates the Homebrew recipe:
+    # https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/pkg-config.rb


Suggested change

# This essentially duplicates the Homebrew recipe:

# https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/pkg-config.rb

# This essentially duplicates the Homebrew recipe

https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/pkg-config.rb no longer exists after Homebrew/homebrew-core#194885

radarhere · 2024-11-16T05:48:28Z

.github/workflows/wheels-dependencies.sh

    build_libpng
    build_lcms2
    build_openjpeg

    ORIGINAL_CFLAGS=$CFLAGS
    CFLAGS="$CFLAGS -O3 -DNDEBUG"
-    if [[ -n "$IS_MACOS" ]]; then
-        CFLAGS="$CFLAGS -Wl,-headerpad_max_install_names"
-    fi


What was your thinking behind removing this? I'm just concerned that #7250 might re-appear.

radarhere · 2024-11-16T06:21:48Z

.github/workflows/wheels-dependencies.sh

+    export PATH="$BUILD_PREFIX/bin:$(dirname $(which python3)):/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin"
+    export CMAKE_PREFIX_PATH=$BUILD_PREFIX
+
+    # Link the brew command into our isolated build directory.


I'm not following this comment. Are you talking about a specific brew command that we're invoking, or just if a user wants to add their own call to brew?

radarhere · 2024-11-16T06:37:44Z

.github/workflows/wheels-dependencies.sh

    build_libwebp
    CFLAGS=$ORIGINAL_CFLAGS

    build_brotli

    if [ -n "$IS_MACOS" ]; then
        # Custom freetype build
-        build_simple freetype $FREETYPE_VERSION https://download.savannah.gnu.org/releases/freetype tar.gz --with-harfbuzz=no
+        build_simple freetype $FREETYPE_VERSION https://download.savannah.gnu.org/releases/freetype tar.gz --without-harfbuzz


https://gitlab.freedesktop.org/freetype/freetype/-/blob/master/builds/unix/configure.raw?ref_type=heads#L429 mentions --with-harfbuzz, but not --without-harfbuzz. What was the thinking here?

radarhere · 2024-11-16T07:18:53Z

.github/workflows/wheels-dependencies.sh

@@ -77,67 +108,88 @@ function build {
    if [ -n "$IS_MACOS" ]; then
        build_simple xorgproto 2024.1 https://www.x.org/pub/individual/proto
        build_simple libXau 1.0.11 https://www.x.org/pub/individual/lib
+        build_simple libXdmcp 1.1.5 https://www.x.org/pub/individual/lib


Adds a build of libXdmcp, to avoid a dependency on the Homebrew-provided version.

I suspect you took out all the brew remove statements, then noticed this was being pulled from brew, and then added it here to replace that.

I'm saying that this wouldn't be part of the Pillow wheels at the moment, so if it's still not after this PR, that sounds fine to me.

This function

Pillow/src/display.c

Line 834 in 5bff2f3

PyImaging_GrabScreenX11(PyObject *self, PyObject *args) {

is the only use of XCB in Pillow, so if it's not necessary, I don't think it needs to be here?

radarhere · 2024-11-16T07:33:56Z

.github/workflows/wheels-dependencies.sh

+        # Homebrew versions of those libraries from /usr/local.
+        build_simple tiff $TIFF_VERSION https://download.osgeo.org/libtiff tar.gz \
+            --with-jpeg-include-dir=$BUILD_PREFIX/include --with-jpeg-lib-dir=$BUILD_PREFIX/lib \
+            --disable-webp --disable-zstd --disable-libdeflate


Suggested change

--disable-webp --disable-zstd --disable-libdeflate

--disable-webp --disable-libdeflate --disable-zstd

Nitpick - use the arguments in the same order as they are mentioned in the comment.

radarhere · 2024-11-16T07:37:14Z

Thanks for your patience. This is getting close - aside from my latest comments, I'm happy with the current state.

Isolate macOS build from Homebrew.

0fe55d6

radarhere changed the title ~~Isolate macOS wheel builds from Homebrew.~~ Isolate macOS wheel builds from Homebrew Oct 25, 2024

radarhere reviewed Oct 25, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

radarhere reviewed Oct 25, 2024

View reviewed changes

.github/workflows/wheels-test.sh Outdated Show resolved Hide resolved

radarhere reviewed Oct 25, 2024

View reviewed changes

freakboy3742 added 3 commits October 25, 2024 10:55

Cleanups and typos identified by code review.

fc35fcc

Tweaks to ensure isolation from Homebrew on x86_64.

00809a2

Bump the multibuild version to fix jpeg-turbo issue.

06dbfed

freakboy3742 requested a review from radarhere October 25, 2024 05:53

Correct a dumb pip invocation error.

5a8373e

hugovk added macOS Build labels Oct 25, 2024

freakboy3742 added 3 commits October 25, 2024 16:04

Explicitly disable libdeflate on libtiff.

140a06e

Possible fix for linux build failures.

0961d3d

Copy manylinux lib64 files from the correct built prefix.

43c34fc

radarhere mentioned this pull request Oct 26, 2024

brew remove libdeflate #8502

Merged

radarhere mentioned this pull request Oct 28, 2024

Set LIBDIR for remaining cmake-built libraries. multi-build/multibuild#538

Merged

radarhere and others added 8 commits October 29, 2024 09:52

Merge branch 'main' into homebrew-isolation

3e4be4b

Revert fribidi/raqm changes for macOS builds.

0855468

Bump multibuild to include more cmake changes.

8308bf3

Correct paths used for Linux build.

c74a5bd

Simplify Linux config by correcting a logic error in macOS config.

72d81e2

Can't check IS_MACOS until common_utils is invoked.

ec214e4

Don't use multibuild variables before invoking multibuild.

d1a4f80

Remove stray debug.

6d13704

Correct handling of vendored fribidi.

e82b539

Merge branch 'main' into homebrew-isolation

904416b

radarhere reviewed Nov 7, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

Correct typo in comment.

4e35852

Co-authored-by: Andrew Murray <[email protected]>

radarhere reviewed Nov 8, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

radarhere reviewed Nov 8, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

Apply suggestions from code review

681a03b

Co-authored-by: Andrew Murray <[email protected]>

radarhere reviewed Nov 12, 2024

View reviewed changes

Disable platform guessing instead of adding dependencies-prefix

378df7a

This was referenced Nov 12, 2024

Disable platform guessing instead of adding dependencies-prefix freakboy3742/Pillow#1

Merged

Having trouble installing libraqm #8232

Closed

radarhere reviewed Nov 13, 2024

View reviewed changes

.github/workflows/wheels-test.sh Outdated Show resolved Hide resolved

freakboy3742 added 2 commits November 13, 2024 11:03

Correct the lookup of libfribidi on x86 macOS installs.

9dc6904

Merge pull request #1 from radarhere/homebrew-isolation

0e3eb70

Disable platform guessing instead of adding dependencies-prefix

radarhere reviewed Nov 15, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

radarhere reviewed Nov 15, 2024

View reviewed changes

.github/workflows/wheels-dependencies.sh Outdated Show resolved Hide resolved

freakboy3742 mentioned this pull request Nov 16, 2024

Add Android Support numpy/numpy#27698

Open

More tweaks from code review.

54f2334

radarhere reviewed Nov 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Isolate macOS wheel builds from Homebrew #8497

Isolate macOS wheel builds from Homebrew #8497

freakboy3742 commented Oct 25, 2024 •

edited

Loading

radarhere Oct 25, 2024

freakboy3742 Oct 25, 2024

radarhere Oct 25, 2024

freakboy3742 commented Oct 25, 2024

freakboy3742 commented Oct 25, 2024

nulano commented Oct 25, 2024

freakboy3742 commented Oct 28, 2024

hugovk commented Oct 28, 2024

nulano commented Nov 4, 2024 •

edited

Loading

freakboy3742 commented Nov 4, 2024

nulano commented Nov 4, 2024

freakboy3742 commented Nov 6, 2024

nulano commented Nov 7, 2024

radarhere Nov 12, 2024

freakboy3742 Nov 13, 2024

radarhere Nov 16, 2024

radarhere Nov 16, 2024

radarhere Nov 16, 2024

radarhere Nov 16, 2024

radarhere Nov 16, 2024

radarhere Nov 16, 2024

radarhere commented Nov 16, 2024

		# This essentially duplicates the Homebrew recipe:
		# https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/pkg-config.rb

	--disable-webp --disable-zstd --disable-libdeflate
	--disable-webp --disable-libdeflate --disable-zstd

Isolate macOS wheel builds from Homebrew #8497

Are you sure you want to change the base?

Isolate macOS wheel builds from Homebrew #8497

Conversation

freakboy3742 commented Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freakboy3742 commented Oct 25, 2024

freakboy3742 commented Oct 25, 2024

nulano commented Oct 25, 2024

freakboy3742 commented Oct 28, 2024

hugovk commented Oct 28, 2024

nulano commented Nov 4, 2024 • edited Loading

freakboy3742 commented Nov 4, 2024

nulano commented Nov 4, 2024

freakboy3742 commented Nov 6, 2024

nulano commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radarhere commented Nov 16, 2024

freakboy3742 commented Oct 25, 2024 •

edited

Loading

nulano commented Nov 4, 2024 •

edited

Loading