Installing a Universal 2 binary

[petrmanek]

Output of brew config

HOMEBREW_VERSION: 4.1.25
ORIGIN: https://github.com/Homebrew/brew
HEAD: c32bd1c7cc0c3b1b914845bddfeda53f4d877a3f
Last commit: 9 days ago
Core tap JSON: 16 Dec 15:35 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 3.1.4 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: quad-core 64-bit skylake
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 14.1.1-x86_64
CLT: 15.0.0.0.1.1694021235
Xcode: 15.0.1

Output of brew doctor

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install yq

Run `brew missing` for more details.

Description of issue

Hi, I have got a question. I am a software engineer attempting to cross-compile my C++ app for both Intel and Apple Silicon, so that I can distribute it as Universal 2 Mach-O binary. I utilize the cmake build system, which can do this by setting the CMAKE_OSX_ARCHITECTURES variable to "x86_64;arm64". Unfortunately, I am running into a small problem with Homebrew and may need your advice.

I use Homebrew to manage my dependencies. As far as I understand, this package manager is capable of detecting CPU architecture of the host and install only the binary suitable to its instruction set. Cmake later finds the installed binary and correctly links against that. This works well when I restrict my target architecture only to the host, however when I attempt to cross-compile the dynamic linker cannot find the other architecture (it cannot find the Intel binary when compiling on M1, and vice versa).

Clearly Homebrew has both binaries available, but only chooses to install one depending on my host CPU. Is there a way to force it to install both? Is there a flag that I can use with brew install or an environment variable that would configure this? Consequently, does that mean that I would need to keep two Cellars, install all dependencies twice, compile my app twice and then concatenate the binaries into the Universal 2 format after the fact? This would seem wasteful, and painfully cumbersome to implement.

Or alternatively, is there a configuration setting that would enable me to install my dependencies already in the Universal 2 format? I have experimented a little bit and found that if I install my dependencies from source as Universal 2 binaries, bypassing Homebrew completely, cmake links them correctly in a single pass without complaining. I am just unhappy that this would effectively lose all the advantages of package management (isolation, updates, dependency tree resolution etc.). If it were possible to achieve the same effect using Homebrew, this would be preferable.

For me this is very much an open question, and I could not find any similar discussions on this topic around here. I would appreciate any ideas or advice you might have!

[SMillerDev]

This is not possible since managing dependencies for distribution outside of Homebrew is not a design goal of Homebrew.

I feel the optimisation we can provide by not making universal binaries and the headaches we avoid by not making universal binaries far outweigh the benefit for this use case that very few users encounter.

[petrmanek]

Yes, I found some discussion on this topic. I understand why it is not worth the hassle, especially considering the main use case for Homebrew and the number of formulae that would need to explicitly opt in to support this. So I am definitely not trying to re-ignite that conversation, rather than that I am just another engineer trying to get a bit more out of his CI. And I would really like to try find a way to convince Homebrew to do this for me, as I found it convenient for many other things in the past.

[kencu]

If you look around the internet a bit you might come across other package managers for MacOS that do attempt to support universal binaries. It's possible one of those might save you some manual labour, although homebrew is by far and away the biggest name in the package manager game on MacOS, and I do use it for many things.

Also, the Apple engineer that is in charge of https://github.com/XQuartz/XQuartz has written a very elegant build system in shell that generates universal binaries for that complex project with many dependencies. You may find that quite adaptable, depending your number of dependencies and tolerance for manually building things.

[petrmanek]

That's a good point, I will try to investigate that. If nothing else, it can be a good inspiration!

[petrmanek]

I have a quick update following several hours of experimentation.

I thought that the easiest way to approach this would be to use brew install --build-from-source to prevent installing a bottled formula and add --interactive to override CFLAGS in order to produce a universal binary instead of a thin one. I tried to do this with zeromq -- one of my dependencies -- and it did not go well. Specifically, I never got the configure command to succeed.

My invocation was simple enough:

./configure CFLAGS="-arch x86_64 -arch arm64" --disable-dependency-tracking

According to AppleClang documentation, if multiple -arch flags are provided the compiler executes multiple times and then uses lipo to concatenate build products into a fat binary. I have verified that this indeed happens with a hello world C program. However, for some reason if autotools run the same test (labeled "checking if compiler can produce executables"), compilation fails with the following message:

clang: unsupported argument "nehalem" to option "-march="

This baffled me as I was not even setting the -march flag anywhere in the code. Checking clang -print-supported-archs showed that nehalem is present in the list, so that was unlikely to be the culprit. Still, it was possible that this flag was passed in through LDFLAGS or other variables autotools use. So, just to be absolutely sure that I can see exactly what the compiler sees I created my own wrapper script, which logs all flags it receives before passing them to the compiler. Turns out that -march is not among them! The invocation is as simple as: clang -arch x86_64 -arch arm64 conftest.c.

After reviewing config.log for an extended period of time, I grew increasingly frustrated with my inability to reproduce the issue. If I make my own conftest.c file and run exactly the same command, I can see a successfully compiled universal binary rather than an error message. Following some more tinkering in the verbose mode, I realized that the clang used by autotools may not necessarily be the same as my clang. After some digging around, I found that this is indeed the case -- Homebrew uses a compiler shim script that intercepts and sometimes modifies various flags, -march not exempt.

I still do not entirely understand how to proceed from here, but I just thought it would be helpful for someone in the future to share my findings. If I am unable to re-configure Homebrew to build a universal binary from source, I may have to resort to plan B. This would mean having two independent Cellars, one for each CPU architecture, and running lipo on pairs of matching files to zip them together into universal binaries. I expect this to be cumbersome and prone to failures, which is why I would prefer to avoid it if possible.

[petrmanek]

I have another update. TL;DR: I managed to get it done.

First things first, I figured out why setting CFLAGS and CXXFLAGS at configure stage was problematic. It turns out that -march=nehalem comes from HOMEBREW_OPTFLAGS, which are injected into CFLAGS by the Homebrew clang shim here. Furthermore, the same script seems to ignore -arch flags unless HOMEBREW_CCCFG contains K. I do not know anything about this particular environment variable, so I did not modify it for good measure. Instead, I discovered that on an adjacent line to HOMEBREW_OPTFLAGS, the same script was extending CFLAGS with the contents of HOMEBREW_ARCHFLAGS variable, which seems to be currently empty in my case.

For that reason, my current solution was as follows:

1. Enter the shell created by brew install --build-from-source --interactive <my_package>
2. Export environment variables: HOMEBREW_OPTFLAGS="" and HOMEBREW_ARCHFLAGS="-arch x86_64 -arch arm64"
3. Run the rest of the formula as expected, in my case: configure, make followed by make install into the provided prefix-
4. Exit the interactive mode and let Homebrew package things up.

I find this nice and elegant, since in step 2 we essentially use the compiler shim, which was previously causing me problems by swallowing some flags, to actually inject the -arch flags that we need. This is done completely out of view of autotools or GNU Make, so then step 3 can be essentially equivalent to building the formula as usual without any patches or exceptions.

Following this I verified that the installed package now contains Universal 2 binaries for both Intel and Apple Silicon:

$ file /usr/local/Cellar/zeromq/4.3.5_1/usr/local/lib/libzmq.5.dylib
/usr/local/Cellar/zeromq/4.3.5_1/usr/local/lib/libzmq.5.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64]
/usr/local/Cellar/zeromq/4.3.5_1/usr/local/lib/libzmq.5.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
/usr/local/Cellar/zeromq/4.3.5_1/usr/local/lib/libzmq.5.dylib (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

My last goal is to automate this process for my CI and scale it up to my other dependencies. Hopefully, they will be as easy to work with as ZeroMQ.

[kencu]

using dual archflags works for some things.

You will find (as I have done it hundreds of times) many other things will not build universal with this simple approach.

I suggest you try macports instead. They have infrastructure to support universal building. It’s not perfect, because supporting universal is hard, but it’s many many hours of work ahead of what you have here, and officially supported there.

[petrmanek]

I will mark this as an answer, since moving to MacPorts effectively solved my issue. For anybody interested, yes -- Homebrew can be convinced universal binaries by playing with environment variables -- but MacPorts is actually designed to do it well.

[petrmanek]

I thought it would be a good idea to check in after a couple weeks. Even though my previously presented approach works for now, I took on board some points made in this thread. Most importantly, that it is a finicky business to convince HB to build universal binaries because that we are bending things out of shape to get the job done, and that while it may be working for now, it may easily break for many reasons in the future. Obviously that would be a dealbreaker for me, as I am trying to set up a somewhat robust CI/CD infrastructure for my project. Long story short, I followed the suggestion made by @kencu and gave MacPorts a try. The bottom line is that I managed to get it done, but I made some observations along the way and encountered a some roadblocks that needed to be sidestepped. I thought this would be a good venue to share them in case future me (or anybody else in a similar position) needs to revisit them.

First, it was a good sign that all my HB packages had their equivalent counterparts in the MacPorts universe. This was not guaranteed, and would be a tremendous pain in the ass if that had not been so. Just like HB allows to enable options during brew install, MacPorts has variants that allow to modify features of the installed package. In my case, the variant I was interested in was +universal (yay, BSD-style flags!). So, my initial attempt was simply to install all my dependencies with that variant enabled.... wrong! Turns out that my dependencies have dependencies, which are resolved automatically, and my desired variants do not propagate to those. This meant that I could install e.g. zmq-devel as +universal but MacPorts would automatically fetch libsodium (a dependency) only for my host architecture. Of course, I would discover this only after installing everything, trying to build my software and seeing that while the first-level dependencies are indeed universal binaries, the linker still complains about not being able to locate the right slices for the rest of my dependencies. As far as I understand, there is no way to propagate a MacPorts variant during the dependency resolution procedure, which would be desirable here, so instead of that I resorted to using a much blunter tool. In /opt/local/etc/macports/variants.conf, one can specify variants that are applied to all installed packages; so I simply added +universal there, and reinstalled all my packages.

Unfortunately that was not the end of it; I had two more setbacks. First happened when I tried to install the qt6 meta-package (one of my dependencies), which contains all the available Qt6 modules. It seems that one of Qt's modules depends on librsvg, which in turn needs rust and cargo as build dependencies. Unfortunately, these packages cannot be installed as +universal because of reasons, so MacPorts called the installation off. I did not really perceive that as a problem because these packages were build dependencies rather than runtime dependencies, meaning that they would be used once to build librsvg on my host machine. As long as they run, and they are capable of cross-compiling librsvg into both architectures I was interested in, they themselves do not really have to be universal. Sadly, there appears to be no facilities in MacPorts to constrain the effects of variants.conf to runtime dependencies only; it is all or nothing. This forced me to work around the problem by identifying the offending Qt module, and installing entire Qt except it. Fortunately, my program did not really need it.

My last problem was with the gnutls package, which again one of the Qt modules depends on. This package was weird -- it advertised support for the +universal variant but when I attempted to install it with that setting enabled, compilation failed. Fishing through the logs to find out what went wrong, I discovered that there were linker messages pointing to missing symbols in libgnu.a (one of its constituent objects), reminding me of this bug which was reported and resolved about a year ago. In any case, I again had to drop the Qt modules that depended on gnutls and pray that my software will not miss them. Following that, there were no further issues. I only had to export CMAKE_PREFIX_PATH to various locations inside /opt/local but short of that, my software is now reliably building for distribution in universal regime! 🎉

I am much happier that I devoted time to resolving this properly. Unlike my previous solution that was based on HB, with MacPorts I did not really need to hack anything or bend anything out of shape to get the job done. This suggests to me that the new solution will be much more robust and sustainable in the long term (famous last words).

Thanks again for your inputs. I hope that this thread helps any other poor soul trying to cross-compile things for universal binaries.

[kencu]

when you install something with macports with the +universal flag, that +universal flag is supposed to be conveyed to all the dependents. If some dependent cannot build +universal and so has no such option, there is supposed to be an error generated to tell you that.

So if you can reproduce your non-propagating +universal variant issue, it would be appropriate for a bug report to be fixed. Something went wrong.

No doubt though, supporting +universal building is somewhat hard -- the successes are always in evolution and improvement.

[scztt]

(late reply, but I stumbled onto this issue and found some of the comments helpful)

You might also try an approach like this:
https://gist.github.com/dyfer/6c83905d4593750105897e51e87ec345

This will pull bottled binaries for both architectures from homebrew to a specific directory, and lipo them together. It was made for a specific and narrow use-case, so I'm sure it will not work universally (no pun intended...), but I think the approach is valid - especially if you have a package with more complex dependencies, where individually dealing with each dependency is not feasible.