cross building from x86_64

[Ludea]

Hi,

I try to build musl toolchain from an x86_64 host to an aarch64 hosts.

My config.mak:

TARGET = aarch64-linux-musl
OUTPUT = /usr/local
BINUTILS_VER = 2.38
GCC_VER = 12.1.0
LINUX_VER = 5.15.44
SHA1_CMD = sha1sum -c
COMMON_CONFIG += CC="aarch64-linux-musl-gcc -static --static" CXX="aarch64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os" CXXFLAGS="-g0 -Os" LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

I get:
configure: error: in `/home/runner/work/musl-cross-make/build/local/aarch64-linux-musl/obj_binutils': configure: error: C compiler cannot create executables
If I delete COMMON_CONFIG += CC="aarch64-linux-musl-gcc -static --static" CXX="aarch64-linux-musl-g++ -static --static" build is successful.

I miss something ? or build is broken with static flag ?

[rofl0r]

not what you're asking, but:

BINUTILS_VER = 2.38
GCC_VER = 12.1.0

you should only use versions that have an associated directory below patches/ , otherwise the compiler created will have all sorts of issues as it will lack critical compatibility patches for musl.

[Ludea]

I revert to use right version, but still have issue

Some logs : https://github.com/Ludea/speedupdate-rs/actions/runs/3205211691/jobs/5237438973

[ryanwoodsmall]

Still have something broken with your config:

2022-10-07T13:47:41.6264418Z make: *** No rule to make target 'linux-5.15.44', needed by 'all'.  Stop.

Remove the LINUX_VER config setting and go with the default here. You're probably still going to hit a cross-compilation problem, though, as what you're doing appears to be more like a "Canadian cross," i.e.:

x86_64 (running on & building for native arch) -> aarch64 (running on native arch, building for foreign arch) -> aarch64 (running on & building for foreign arch)

There's a long-standing issue with GCC that has been surfaced on (at least) canadian-cross aarch64 builds. A link to an old GCC mention - this exhibits as undefined references to host_detect_local_cpu and I've hit it on GCC 9.x:

• https://linaro-toolchain.linaro.narkive.com/mealwez3/cross-compilation-issue#post1
• How to change dynamic links into static links? #153 (comment)

The __aarch64__ symbol appears to be defined in the build for that target platform, at least on versions <10, regardless of the host platform, which (I believe) causes native CPU detection to be attempted when running the aarch64 compiler on x86_64, which causes the CPU detection routine to not be defined/found, since it's not compiling native code. I'm not sure if this is still broken on GCC 11 or 12, but IIRC I hit it on the first 10.x release as well. Easiest way to avoid that issue is to patch the middle build (native x86_64 compiler binaries targeting aarch64) with a patch like so - this is for 9.4.0, different versions of GCC will vary:

# cat patches/gcc-9.4.0/1111-no-__aarch64__-symbol.diff.OFF 
--- a/gcc/config/aarch64/aarch64.h      2022-10-07 19:58:21.284795937 +0000
+++ b/gcc/config/aarch64/aarch64.h      2022-10-07 19:58:32.580763886 +0000
@@ -1083,7 +1083,7 @@
 #define MCPU_TO_MARCH_SPEC_FUNCTIONS \
   { "rewrite_mcpu", aarch64_rewrite_mcpu },
 
-#if defined(__aarch64__)
+#if defined(NOT__aarch64__)
 extern const char *host_detect_local_cpu (int argc, const char **argv);
 #define HAVE_LOCAL_CPU_DETECT
 # define EXTRA_SPEC_FUNCTIONS                                          \

Remove this patch for the final native aarch64 toolchain. If it's not removed for the final round of compilation, I'm not sure what the resulting compiler will do when mtune/march/etc. is set to native/local CPU.

This leads to the next issue: during compilation of the final native aarch64 compiler, the cross-compiler running on x86_64 and targeting aarch64 will possibly (likely?) generate code that needs to run on x86_64 but is compiled for aarch64. Unless qemu-user is installed and working, this will probably fail. It'll be slow even if it works.

Finally, you probably shouldn't mix binaries for different architectures in the same installation directory. While it shouldn't hurt anything, it's asking for trouble.

Here's how I'd go about achieving what you're aiming for here. Note that you will probably need the qemu-user-static package installed and working on Debian or Ubuntu for this...

• 0: compile a "bootstrap" native compiler with system compilers, running on & targeting x86_64
• 1: compile a native x86_64 musl compiler with the bootstrap compiler built in step 0, running on & targeting x86_64
• 2: install patch to disable the __aarch64__ symbol for GCC
• 3: compile a musl aarch64 cross compiler with the native x86_64 musl compiler built in step 1, running on x86_64 & targeting aarch64
• 4: remove the patch installed in step 2
• 5: compiler a musl native aarch64 compiler targeting aarch64 with the cross-compiler built in step 3, running on & targeting aarch64

config.mak snippets for each of the compiler building steps, dropping stuff into /usr/local/musl per-arch - remove ccache if you're not going to be re-running compiles; it significantly speeds up the first (native) pass if you'll run it multiple times, but will slow down same if you're only running it once.

native x86_64 bootstrap musl compiler

# 0_config.mak.bootstrap 
TARGET = x86_64-linux-musl
OUTPUT = /usr/local/musl/bootstrap
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache gcc -static --static"
COMMON_CONFIG += CXX="ccache g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

native x86_64 musl compiler

# 1_config.mak.native-x86_64 
TARGET = x86_64-linux-musl
OUTPUT = /usr/local/musl/x86_64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/bootstrap/bin/x86_64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/bootstrap/bin/x86_64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

native musl cross compiler targeting aarch64

2. Install the patch for gating the __aarch64__ symbol here...

# 3_config.mak.cross-aarch64 
TARGET = aarch64-linux-musl
OUTPUT = /usr/local/musl/x86_64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/x86_64/bin/x86_64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/x86_64/bin/x86_64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

4. Remove the patch for gating the __aarch64__ symbol here.

native aarch64 musl compiler

# 5_config.mak.native-aarch64 
TARGET = aarch64-linux-musl
OUTPUT = /usr/local/musl/aarch64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/x86_64/bin/aarch64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/x86_64/bin/aarch64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

This will drop compilers into their own directories - /usr/local/musl/bootstrap, /usr/local/musl/x86_64 and /usr/local/musl/aarch64. The bootstrap step isn't really necessary, but I'd still recommend doing it just to guarantee you can compile a compiler that can compile itself or something similar.

Note that I can't guarantee that any of this will work, nor can I say the middle aarch64 or resultant aarch64 compiler running on and targeting aarch64 is 100% trustable. For GCC at least, compiling natively, or at least on the same family of CPU (i.e., i686->x86_64, armhf->aarch64, etc.), is still the recommended approach I believe.

[zv-io]

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

[Ludea]

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

Yes, I started with your fork.
I also try to download your binaries with wget, without success, so I try to clone and build both your fork and this repo

[zv-io]

Yes, I started with your fork. I also try to download your binaries with wget, without success, so I try to clone and build both your fork and this repo

Are you trying to download them from inside of GitHub Actions or CI pipelines? See here: https://github.com/orgs/community/discussions/27906

[Ludea]

I try from Github action

[Ludea]

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

I'm interresting about some instructions to build it from Ubuntu ;-)

Are you trying to download them from inside of GitHub Actions or CI pipelines? See here: https://github.com/orgs/community/discussions/27906

I fix it, by upload your archive to a static file provider and download the toolchain from it