Conversation
HIP graph replay of XLA command buffers segfaults inside the HSA runtime's InterceptQueue on current ROCm stacks -- deterministically, on the first execution of a while-loop command buffer, after compilation has succeeded. XLA's own escape hatch is `--xla_gpu_enable_command_buffer=` (the empty set); apply that as the default for ROCm clients by clearing the command-buffer type list in the debug options Reactant builds per compile. An explicit `xla_debug_options=(; xla_gpu_enable_command_buffer=...)` still wins, so re-enabling for testing newer ROCm stacks stays a one-liner. This costs real performance on loop-heavy workloads (full per-launch overhead each iteration), but a deterministic crash on the first executed step costs more. Reported by Alex Zinenko (@ftynse) on gfx950 / ROCm 7.1: https://gist.github.com/ftynse/5c3816a1cea5daa7dc189b937b62f9ca (issue 6) Co-authored-by: Alex Zinenko <ftynse@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015drPT8jCwS74HnX6vEBCxz
Member
Author
|
Upstream issue filed: openxla/xla#46952 — once resolved there, reverting this default is a one-liner (drop the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mitigates issue 6 of @ftynse's ROCm findings:
SIGSEGVinrocr::core::InterceptQueue::Submit→RocmCommandBuffer::LaunchGraphon the first execution of a while-loop command buffer, on gfx950 / ROCm 7.1 — deterministic, after compile/spinup succeeded, and gone entirely with command buffers off (theirsmoke08vssmoke09runs).Change
XLA.get_debug_options/make_compile_optionsnow take the client'splatform; on"rocm"thexla_gpu_enable_command_bufferlist is cleared — the per-compile equivalent of XLA's own documented escape hatch--xla_gpu_enable_command_buffer=. Both compile paths (PJRTcompile_mlir!and the IFRT resharding path) pass the platform through.The override ordering keeps this a default, not a lock-out: user kwargs are applied after, so
xla_debug_options=(; xla_gpu_enable_command_buffer=[...])re-enables them — the right knob for testing whether a newer ROCm stack still crashes.Cost
Real and worth stating: a time-stepping loop is many small kernels, exactly the shape command buffers accelerate — disabling them pays full per-launch overhead every iteration. But a deterministic crash on the first executed step costs more, and the flag flip back is one line once the HSA/rocprofiler/XLA interaction is isolated upstream.
Verification
Unit-checked on CUDA hardware (no AMD GPU available locally):
platform="rocm"produces an empty command-buffer list,platform="cuda"keeps XLA's defaults (CONDITIONAL, CUBLAS, …, FUSION), an explicit user override wins over the ROCm default, and an end-to-end@jitcompile on CUDA is unaffected.🤖 Generated with Claude Code
https://claude.ai/code/session_015drPT8jCwS74HnX6vEBCxz