Skip to content

Build ROCm against TheRock 10.0.0, the ROCm XLA uses - #237

Merged
wsmoses merged 1 commit into
mainfrom
pb/rocm-10
Oct 11, 2026
Merged

wsmoses merged 1 commit into
mainfrom
pb/rocm-10

Conversation

@wsmoses

@wsmoses wsmoses commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Enzyme-JAX's JAX bump (EnzymeAD/Enzyme-JAX#3368) fails the ROCm Reactant_jll build. The new XLA builds its MORI collective kernels (xla/backends/gpu/collectives/mori_kernels.cu.cc, under if_rocm_is_configured), and MORI calls __builtin_amdgcn_global_load_b128:

external/roc_mori/include/mori/collective/XLA/collectives_common.hpp:128:14: error: use of undeclared identifier '__builtin_amdgcn_global_load_b128'

Why a ROCm bump:

  • Where the builtin lives: it is only in AMD's downstream clang; upstream LLVM has no such builtin. Our "7.1" toolchain is the TheRock nightly 7.10.0a20251103, whose clang doesn't have it.
  • What XLA uses: XLA's hermetic ROCm is now the TheRock multiarch 10.0.0 release (third_party/gpus/rocm/rocm_redist.bzl, same sha256 as here).
  • Checked locally: its clang (AMD clang 23.0.0git) compiles the builtin to global_load_dwordx4 / global_load_b128 for every target in TF_ROCM_AMDGPU_TARGETS: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1200 and gfx1201.

Changes:

  • Source: therock-dist-linux-multiarch-10.0.0.tar.gz, replacing the gfx120X nightly.
  • Platform tag: gpu_version becomes 10.0 instead of 7.1, in the build matrix and in platform_augmentation.jl, which picks the ROCm build when an AMD driver is present. Reactant.jl doesn't refer to the tag.
  • Sonames: the nightly-specific renames (*.so.7.1.25442-19ae9ff849) become a pattern that strips TheRock's build suffix. 10.0 names them *.so.7.15.26333-0000000. I checked the loop on the 10.0 files: the .so.7 links resolve.
  • Clang resource directory: 10.0 ships clang/23; 7.1 shipped clang/22.
    • The cuda_wrappers overrides now read the version from the tarball.
    • Enzyme-JAX's workspace.bzl patches XLA's ROCm crosstool with a hardcoded lib/llvm/lib/clang/22/include: rocm_configure.bzl takes its realpath, and the crosstool declares it to the inclusion check. The first CI run failed there (/workspace/srcdir/lib/llvm/lib/clang/22 (No such file or directory)). The recipe now symlinks clang/22 → clang/23 and spells hipcc's -isystem paths through clang/22, so both declared spellings match what the compiler reports. Checked on the 10.0 tree: clang/22/include resolves to clang/23/include, including stddef.h and cuda_wrappers.
  • amdgcn link: 10.0 already ships amdgcn -> lib/llvm/amdgcn. The recipe's own ln -s followed it and created lib/llvm/amdgcn/amdgcn, a link back to its own directory, so the second CI run failed globbing rocm_dist/amdgcn/** (Symbolic link loop). That ln is removed.
  • Removed libraries: 10.0 has no libhipsolver_fortran and no libamd_comgr_loader (libamd_comgr.so.3 is still there). Their installs and the libhipsolver_fortran product are dropped; Reactant.jl uses neither. Every other installed library and product exists in 10.0.

Not verified here: the full BinaryBuilder build; that's what this PR's CI runs. Changing the ROCm toolchain also invalidates the cached host tools on that job, so it will hit the __throw_bad_array_new_length failure until #236 is merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD

XLA now builds its MORI collective kernels for ROCm, which call
__builtin_amdgcn_global_load_b128, a builtin of AMD's downstream clang
the 7.10.0a20251103 nightly does not have. Move to the TheRock multiarch
10.0.0 release XLA itself pins (third_party/gpus/rocm/rocm_redist.bzl),
whose clang has it for every target we build.

- The platform tag follows: gpu_version 10.0 in place of 7.1, in the
  build matrix and in the runtime platform augmentation.
- The soname suffix is stripped by pattern, not by the nightly's exact
  version, and the clang resource directory is read from the tarball.
- Enzyme-JAX's patch to XLA's ROCm crosstool hardcodes clang/22/include
  (the resource directory it declares to the inclusion check); 10.0
  ships clang 23, so alias clang/22 to it and point hipcc's -isystem
  paths through the alias.
- 10.0 ships amdgcn -> lib/llvm/amdgcn itself; creating it again made
  ln follow the link and nest a self-loop (lib/llvm/amdgcn/amdgcn),
  which breaks the rocm repository's amdgcn/** glob. Drop that ln.
- 10.0 ships neither libhipsolver_fortran nor libamd_comgr_loader; drop
  their installs and the libhipsolver_fortran product.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
@wsmoses
wsmoses merged commit 6d03664 into main Oct 11, 2026
16 checks passed
@wsmoses
wsmoses deleted the pb/rocm-10 branch October 11, 2026 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant