Cannot cross-compile rust while providing pre-built LLVM

[rhelmot]

I'm a little lost at how exactly to report this because I don't fully understand the error - I've spent several hours trying to reverse engineer the build process and the following is all I really have to show for it:

I'm trying to simplify the nixpkgs build of rustc, and I'm getting failures in cross configurations (linux x86_64->aarch64 as my testcase). This build configuration requires using pre-built LLVMs and providing the llvm-config binary for each target in a cross build. I'm getting build failures when linking the stage2:x86_64-unknown-linux-gnu -> stage3:aarch64-unknown-linux-gnu rust-analyzer-proc-macro-srv, claiming that it can't find -lLLVM-19. Before this, it was successfully able to link the stage1:x86_64-unknown-linux-gnu -> stage2:aarch64-unknown-linux-gnu rustc_driver against libLLVM-19.so for the aarch64 platform without any issue.

I was able to determine that it's deciding it needs to link with -lLLVM-19 because of some data in the librustc_llvm.rmeta file - when I edited the rmeta file to say LLBM instead, the error message changed accordingly but still failed in the same spot. Based on this, it seems like the underlying bug is that the corresponding -L path flags aren't being propagated even though the -l flags are.

However, I haven't been able to determine why it does work in the non-cross case. I give it equal odds my hypothesis is just wrong vs there is some special cross/non-cross logic in the rustc build which is messing this up.

Note: I tried jamming the is_cross predicate in the rustc_llvm build script to false, since the comments around it seemed to not match super well for my nix use-case, but it failed the same way.

Here's the config.toml (gzip'd to make github happy) for my build

Here's the build log for the failing build

llvm-config output:

[+] ~/proj/nix/nixpkgs% /nix/store/6b44i6knzdd71qkjygd8gjwzq00x3ipl-llvm-aarch64-unknown-linux-gnu-19.1.6-dev/bin/llvm-config-native --ldflags
-L/nix/store/5bck4qix6xd3hf4rkjfxcga6w4n8ks6y-llvm-aarch64-unknown-linux-gnu-19.1.6-lib/lib 
[+] ~/proj/nix/nixpkgs% /nix/store/6b44i6knzdd71qkjygd8gjwzq00x3ipl-llvm-aarch64-unknown-linux-gnu-19.1.6-dev/bin/llvm-config-native --libs
-lLLVM-19
[+] ~/proj/nix/nixpkgs% /nix/store/dqq9r3nggw20fqs3lsyci7m5pskqp7xh-llvm-19.1.6-dev/bin/llvm-config --ldflags
-L/nix/store/7g1gw5zkssblpcbgicnnbvwpbmr4za55-llvm-19.1.6-lib/lib 
[+] ~/proj/nix/nixpkgs% /nix/store/dqq9r3nggw20fqs3lsyci7m5pskqp7xh-llvm-19.1.6-dev/bin/llvm-config --libs
-lLLVM-19

If anyone would like to reproduce this with nix, you can do so with nix build github:rhelmot/nixpkgs/04a5a2d2fb5216c0f401529d08a1c034fcbd890d#pkgsCross.aarch64-multiplatform.rustc.unwrapped. It's based on a staging branch of nixpkgs, so it will have to rebuild the world and will take a few hours.

Lemme know if there's anything else I can provide.

[jyn514]

you have provided llvm-config flags for the host, but not for the aarch64 target. can you post the output of /nix/store/6b44i6knzdd71qkjygd8gjwzq00x3ipl-llvm-aarch64-unknown-linux-gnu-19.1.6-dev/bin/llvm-config-native --ldflags --libs?

However, I haven't been able to determine why it does work in the non-cross case. I give it equal odds my hypothesis is just wrong vs there is some special cross/non-cross logic in the rustc build which is messing this up.

non-cross builds use a different version of llvm, one that's compiled for the host and not the target. note that you have this output in your build log:

 "-L" "/build/rustc-1.83.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib"

i'm not entirely why rustc is trying to link an artifact against llvm. my best guess is that proc-macro-srv is weird because it links against rustc and therefore needs to link against rustc's dynamic libraries. you could try passing --exclude proc-macro-srv and see if this error happens for any other artifact.

can you post the output of ls /build/rustc-1.83.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib? i think it will be empty; i think what's happened is that bootstrap respects target.aarch64.llvm-config when building rustc itself, but something goes wrong when rustc is compiling an artifact for that target.