Offload bootstrap #148671

ZuseZ4 · 2025-11-07T21:24:22Z

The recommended way to build the offload project changed.
While already at it, testing how much overhead we get from building (or shipping) the two runtimes.

r? @Kobzol

rustbot · 2025-11-07T21:24:25Z

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

ZuseZ4 · 2025-11-07T21:26:41Z

@bors2 try @rust-timer queue

(@Kobzol feel free to change if you want to see something else).

rust-timer · 2025-11-07T21:26:44Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-07T22:26:45Z

💔 Test for 7069335 failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-07T23:05:23Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-07T23:05:26Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-07T23:19:09Z

💔 Test for 31000c9 failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-08T02:54:18Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-08T02:54:21Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-16T03:04:31Z

💔 Test for 57b6a7b failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-16T08:20:25Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-16T08:20:27Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-16T09:36:39Z

💔 Test for 3b40cbb failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-16T13:34:17Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-16T13:34:19Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-16T13:49:10Z

💔 Test for 67eb183 failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-16T14:33:56Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-16T14:33:59Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-bors · 2025-11-16T14:49:33Z

💔 Test for cf671bb failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-17T00:24:44Z

@bors2 try @rust-timer queue

rust-timer · 2025-11-17T00:24:46Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

Offload bootstrap

rust-log-analyzer · 2025-11-17T00:37:35Z

The job dist-x86_64-linux failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 32.64s
##[endgroup]
[2025-11-17T00:37:13.428Z INFO  opt_dist::timer] Section `Stage 1 (Rustc PGO)` starts
[2025-11-17T00:37:13.428Z INFO  opt_dist::timer] Section `Stage 1 (Rustc PGO) > Build PGO instrumented rustc and LLVM` starts
[2025-11-17T00:37:13.428Z INFO  opt_dist::exec] Executing `RUST_BACKTRACE=full python3 /checkout/x.py build --target x86_64-unknown-linux-gnu --host x86_64-unknown-linux-gnu --stage 2 library/std --set rust.llvm-bitcode-linker=false --set build.extended=false --set rust.codegen-backends=['llvm'] --set rust.deny-warnings=false --rust-profile-generate /tmp/tmp-multistage/opt-artifacts/rustc-pgo --set llvm.thin-lto=false --set llvm.link-shared=true [at /checkout/obj]`
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.06s
##[endgroup]
[TIMING:start] compile::Assemble { target_compiler: Compiler { stage: 2, host: x86_64-unknown-linux-gnu, forced_compiler: false } }
[TIMING:start] builder::Libdir { compiler: Compiler { stage: 2, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu }
---
CMAKE_x86_64-unknown-linux-gnu = None
CMAKE_x86_64_unknown_linux_gnu = None
HOST_CMAKE = None
CMAKE = None
running: cd "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build" && CMAKE_PREFIX_PATH="" DESTDIR="" LC_ALL="C" "cmake" "/checkout/src/llvm-project/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_UNREACHABLE_OPTIMIZE=OFF" "-DLLVM_ENABLE_PLUGINS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;AMDGPU;ARM;BPF;Hexagon;LoongArch;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR;M68k;CSKY;Xtensa" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=36" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_ENABLE_WARNINGS=OFF" "-DLLVM_INSTALL_UTILS=ON" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DRUNTIMES_CMAKE_ARGS=-DLIBOMPTARGET_PLUGINS_TO_BUILD=amdgpu;nvptx" "-DLLVM_ENABLE_PROJECTS=lld" "-DLLVM_ENABLE_RUNTIMES=compiler-rt;offload;openmp" "-DLLVM_VERSION_SUFFIX=-rust-1.93.0-nightly" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER_LAUNCHER=sccache" "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache" "-DCMAKE_C_COMPILER=clang" "-DCMAKE_CXX_COMPILER=clang++" "-DCMAKE_ASM_COMPILER=clang" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu -fdebug-prefix-map=/checkout=/rustc/llvm" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu -fdebug-prefix-map=/checkout=/rustc/llvm" "-DCMAKE_AR=/rustroot/bin/llvm-ar" "-DCMAKE_RANLIB=/rustroot/bin/llvm-ranlib" "-DCMAKE_SHARED_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DCMAKE_MODULE_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DCMAKE_EXE_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DLLVM_ENABLE_ZSTD=FORCE_ON" "-DLLVM_USE_STATIC_ZSTD=TRUE" "-DCMAKE_INSTALL_PREFIX=/checkout/obj/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is Clang 21.1.0
-- The CXX compiler identification is Clang 21.1.0
-- The ASM compiler identification is Clang
-- Found assembler: /rustroot/bin/clang
-- Detecting C compiler ABI info
---
CMake Error at /usr/local/share/cmake-3.20/Modules/ExternalProject.cmake:3113 (get_property):
  get_property could not find TARGET clang-resource-headers.  Perhaps it has
  not yet been created.
Call Stack (most recent call first):
  /usr/local/share/cmake-3.20/Modules/ExternalProject.cmake:3238 (_ep_get_file_deps)
  /usr/local/share/cmake-3.20/Modules/ExternalProject.cmake:3686 (_ep_add_configure_command)
  cmake/modules/LLVMExternalProjectUtils.cmake:381 (ExternalProject_Add)
  runtimes/CMakeLists.txt:85 (llvm_ExternalProject_Add)
  runtimes/CMakeLists.txt:163 (builtin_default_target)


-- Configuring incomplete, errors occurred!
See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeOutput.log".
See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeError.log".
---
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:834:13
  12:     0x55d2574cebda - std::panicking::panic_handler::{{closure}}::hddd6216c3fddee3b
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:707:13
  13:     0x55d2574c8c39 - std::sys::backtrace::__rust_end_short_backtrace::h5e7a56f283378525
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/sys/backtrace.rs:174:18
  14:     0x55d2574ad18d - __rustc[256c90bb7e548b45]::rust_begin_unwind
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:698:5
  15:     0x55d2566eadc0 - core::panicking::panic_fmt::hf19c198748ef6a95
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/core/src/panicking.rs:80:14
  16:     0x55d256fce38b - cmake::fail::h5ffd31797e7666dd
  17:     0x55d256fce012 - cmake::run::h9db521ef9ab4f16f
---
  32:     0x55d2566ec2f6 - std::sys::backtrace::__rust_begin_short_backtrace::h27046ffc55333d92
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/sys/backtrace.rs:158:18
  33:     0x55d2566ec2d9 - std::rt::lang_start::{{closure}}::h987ce18149d64279
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/rt.rs:206:18
  34:     0x55d2574bc1c0 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hdf957eac011d0136
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/core/src/ops/function.rs:287:21
  35:     0x55d2574bc1c0 - std::panicking::catch_unwind::do_call::h9ff550f7799115e1
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:590:40
  36:     0x55d2574bc1c0 - std::panicking::catch_unwind::ha7ce46787bb12905
                               at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:553:19
---
[2025-11-17T00:37:20.066Z INFO  opt_dist::utils] Free disk space: 1.27 TiB out of total 2.18 TiB (41.63% used)
Error: Optimized build pipeline has failed

Caused by:
    Command RUST_BACKTRACE=full python3 /checkout/x.py build --target x86_64-unknown-linux-gnu --host x86_64-unknown-linux-gnu --stage 2 library/std --set rust.llvm-bitcode-linker=false --set build.extended=false --set rust.codegen-backends=['llvm'] --set rust.deny-warnings=false --rust-profile-generate /tmp/tmp-multistage/opt-artifacts/rustc-pgo --set llvm.thin-lto=false --set llvm.link-shared=true [at /checkout/obj] has failed with exit code Some(1)

Stack backtrace:
   0: <anyhow::Error>::msg::<alloc::string::String>
             at /rust/deps/anyhow-1.0.99/src/backtrace.rs:27:14
   1: <opt_dist::exec::CmdBuilder>::run
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/exec.rs:80:17
   2: <opt_dist::exec::Bootstrap>::run
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/exec.rs:199:18
   3: opt_dist::execute_pipeline::{closure#1}::{closure#0}
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/main.rs:256:21
   4: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}::{closure#0}, ()>
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/timer.rs:111:22
   5: opt_dist::execute_pipeline::{closure#1}
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/main.rs:245:15
   6: <opt_dist::timer::TimerSection>::section::<opt_dist::execute_pipeline::{closure#1}, opt_dist::training::RustcPGOProfile>
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/timer.rs:111:22
   7: opt_dist::execute_pipeline
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/main.rs:242:35
   8: opt_dist::main
             at /rustc/f795c48bdf10a181766ac0d13ad9aed492bbcab0/src/tools/opt-dist/src/main.rs:467:18
   9: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/core/src/ops/function.rs:250:5
  10: std::sys::backtrace::__rust_begin_short_backtrace::<fn() -> core::result::Result<(), anyhow::Error>, core::result::Result<(), anyhow::Error>>
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/sys/backtrace.rs:158:18
  11: std::rt::lang_start::<core::result::Result<(), anyhow::Error>>::{closure#0}
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/rt.rs:206:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/core/src/ops/function.rs:287:21
  13: std::panicking::catch_unwind::do_call
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:590:40
  14: std::panicking::catch_unwind
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:553:19

rust-bors · 2025-11-17T00:37:44Z

💔 Test for f795c48 failed: CI. Failed jobs:

try - dist-x86_64-linux (web logs, extended logs)

ZuseZ4 · 2025-11-19T09:23:44Z

@Kobzol You mentioned before that we build llvm here like 5 times, and I'm indeed getting a bit confused about whether I am changing the right things.

The way I see it from the logs (e.g. #148671 (comment)) is

We build clang-21.1.0/lld/llvm/compiler-rt (a minimal subset without all the extras user like I might request via bootstrap.toml)
We use the new clang to build llvm/clang-21.1.5, but not yet lld
We use the clang to build runtimes, in my case offload/openmp
We fail, because our new clang uses ld instead of lld (which is only used by default by our previous clang-21.1.0), and ld errors with Could NOT find Threads (missing: Threads_FOUND).

I see that despite enabling lld in bootstrap.toml, we don't build the lld project anymore. Is that intended here, or am I misinterpreting the logs? As you can see in the commit history, I tried to explicitly enable lld as project in one commit.
I tried a few flags, to get our new clang-21.1.5 to not use ld, but instead the lld build as part of clang-21-1.0, but without success.

I know bootstrap is a bit tricky, but can you give me a vague explanation of what's the goal of these steps here, so I can guess better what to change?

2025-11-15T23:50:15.0639979Z running: cd "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build" && CMAKE_PREFIX_PATH="" DESTDIR="" LC_ALL="C" "cmake" "/checkout/src/llvm-project/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_UNREACHABLE_OPTIMIZE=OFF" "-DLLVM_ENABLE_PLUGINS=OFF" "-DLLVM_TARGETS_TO_BUILD=AArch64;AMDGPU;ARM;BPF;Hexagon;LoongArch;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=AVR;M68k;CSKY;Xtensa" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_INCLUDE_BENCHMARKS=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_BINDINGS=OFF" "-DLLVM_ENABLE_Z3_SOLVER=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=36" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu" "-DLLVM_ENABLE_WARNINGS=OFF" "-DLLVM_INSTALL_UTILS=ON" "-DLLVM_ENABLE_ZLIB=ON" "-DLLVM_LINK_LLVM_DYLIB=ON" "-DLLVM_ENABLE_LIBXML2=OFF" "-DRUNTIMES_CMAKE_ARGS=-DLIBOMPTARGET_PLUGINS_TO_BUILD=amdgpu;nvptx" "-DLLVM_ENABLE_PROJECTS=clang" "-DLLVM_ENABLE_RUNTIMES=compiler-rt;offload;openmp" "-DLLVM_VERSION_SUFFIX=-rust-1.93.0-nightly" "-DCMAKE_INSTALL_MESSAGE=LAZY" "-DCMAKE_C_COMPILER_LAUNCHER=sccache" "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache" "-DCMAKE_C_COMPILER=clang" "-DCMAKE_CXX_COMPILER=clang++" "-DCMAKE_ASM_COMPILER=clang" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu -fdebug-prefix-map=/checkout=/rustc/llvm" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu -fdebug-prefix-map=/checkout=/rustc/llvm" "-DCMAKE_AR=/rustroot/bin/llvm-ar" "-DCMAKE_RANLIB=/rustroot/bin/llvm-ranlib" "-DCMAKE_SHARED_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DCMAKE_MODULE_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DCMAKE_EXE_LINKER_FLAGS= -Wl,-Bsymbolic -static-libstdc++" "-DLLVM_ENABLE_ZSTD=FORCE_ON" "-DLLVM_USE_STATIC_ZSTD=TRUE" "-DCMAKE_INSTALL_PREFIX=/checkout/obj/build/x86_64-unknown-linux-gnu/llvm" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 --target=x86_64-unknown-linux-gnu" "-DCMAKE_BUILD_TYPE=Release"

ZuseZ4 added 2 commits November 7, 2025 16:12

update offload build steps

e668803

I surely picked the wrong runner again, right?

861cb3b

rustbot assigned Kobzol Nov 7, 2025