Skip to content

Raising: dead lanes must not scatter at all - #2993

Merged
wsmoses merged 1 commit into
mainfrom
pb/masked-scatter-dead-lanes
Sep 10, 2026
Merged

wsmoses merged 1 commit into
mainfrom
pb/masked-scatter-dead-lanes

Conversation

@wsmoses

@wsmoses wsmoses commented Aug 27, 2026

Copy link
Copy Markdown
Member

A store under a lane mask raises as gather-select-scatter, writing the original value back on masked-out lanes. But a dead lane's index expression is unconstrained — strip-mine bound clamps (max(extent, tid + bd)) collapse dead lanes onto live lanes' slots — and scatter applies duplicate indices in unspecified order, so the stale write-back can land after the live lane's update and clobber it.

Found via mfem's QuadratureInterpolator under the XLA backend (#2968): the dynamic 2D tensor-product eval kernel produced partially-zero, element-varying results; a PJRT harness on the raised module reproduced it deterministically, and redirecting dead lanes out of bounds fixes it bit-exactly.

The fix selects dead lanes' indices to -1; out-of-bounds scatter updates are dropped, so masked-out lanes write nothing. This also removes the gather of the original values.

This PR now carries only the raising change. The enzyme-hlo-opt half that resolves the masked index ahead of the single-point scatter-to-DUS rewrite (ScatterMaskedIndexSimplify) is split out into its own PR, and the bounds guard on that rewrite is #3022. Ordering matters: with this change a masked-out lane's correctness rests on the scatter dropping its out-of-bounds index, so the DUS rewrite must not clamp it. This should land with, or after, one of those two.

The previous form let enzyme-hlo-opt fold a full-grid masked store into a pure select; with the index-select form the scatter survives in affine_to_stablehlo_masked_scatter.mlir. Teaching the dense-scatter fold to look through the index select would recover that.

🤖 Generated with Claude Code

https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD

@wsmoses
wsmoses requested review from Pangoraw and avik-pal August 27, 2026 16:48
@wsmoses

wsmoses commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

@avik-pal @Pangoraw can I get a proper review/in depth audit of this [unreviewed by me so far to be clear] vibed up PR.

In particular one bad consequence is that this breaks the scatter removal so we may need to update things

@wsmoses

wsmoses commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

Pushed a required companion fix: the single-point scatter→dynamic_update_slice rewrite in enzyme-hlo-opt was unsound against the new out-of-bounds dead-lane indices — scatter drops OOB updates but DUS clamps the index back in range, so a masked-off store corrupted slot zero (caught by mfem's PAIdentityInterp, isolated with a buffer-level runtime diff + PJRT harness oracle). A new always-on pattern resolves the mask before those rewrites: scatter at the live index and write the original value back when masked off (a no-op store), which then lowers to a sound DUS. Lit test included.

Comment on lines +1177 to +1181
SmallVector<int64_t> idxMaskDims;
for (int64_t k = 0, e = (int64_t)gridShape.size(); k < e; ++k)
idxMaskDims.push_back(k);
Value idxMask = stablehlo::BroadcastInDimOp::create(
builder, loc, idxMaskTy, broadcastedMask, idxMaskDims);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this can just be a reshape (since there should be a single additional trailing 1 element dim)

Pangoraw

This comment was marked as resolved.

wsmoses added a commit that referenced this pull request Sep 5, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).
wsmoses added a commit that referenced this pull request Sep 5, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
wsmoses added a commit that referenced this pull request Sep 6, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).
wsmoses added a commit that referenced this pull request Sep 6, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
wsmoses added a commit that referenced this pull request Sep 6, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).
wsmoses added a commit that referenced this pull request Sep 6, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
@wsmoses
wsmoses changed the base branch from main to pb/scatter-mask-reduce September 7, 2026 03:59
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from 7912307 to 69f03fd Compare September 7, 2026 03:59
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 1c43472)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit 2696b7d)
@wsmoses
wsmoses force-pushed the pb/scatter-mask-reduce branch 2 times, most recently from 10e578e to e297f58 Compare September 7, 2026 16:59
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from 69f03fd to f777805 Compare September 7, 2026 17:03
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 1c43472)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit 2696b7d)
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from e189479 to 14b40aa Compare September 7, 2026 19:35
@wsmoses
wsmoses force-pushed the pb/scatter-mask-reduce branch from eb816b7 to d7576d1 Compare September 7, 2026 19:39
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from 14b40aa to ce1ad00 Compare September 7, 2026 19:41
Base automatically changed from pb/scatter-mask-reduce to main September 7, 2026 19:47
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit aecfaf7009078d2e59f497733af9759e6d04f17c)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit 78be8e326f43b77003cc5bc69e687dd1e3e63b23)
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from ce1ad00 to 9c4c1d5 Compare September 7, 2026 19:50
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 4d9d037)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit d483774)
wsmoses added a commit that referenced this pull request Sep 7, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 4d9d037)
wsmoses added a commit that referenced this pull request Sep 7, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit d483774)
wsmoses added a commit that referenced this pull request Sep 8, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 4d9d037)
wsmoses added a commit that referenced this pull request Sep 8, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit d483774)
wsmoses added a commit that referenced this pull request Sep 8, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 4d9d037)
wsmoses added a commit that referenced this pull request Sep 8, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit d483774)
wsmoses added a commit that referenced this pull request Sep 8, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
wsmoses added a commit that referenced this pull request Sep 8, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
@wsmoses
wsmoses force-pushed the pb/masked-scatter-dead-lanes branch from 9c4c1d5 to 33a1e1e Compare September 8, 2026 02:14
wsmoses added a commit that referenced this pull request Sep 8, 2026
Squash of pb/dead-lane-scatter @ 7912307 onto union2 (3 commits: the
dead-lane scatter form, the tridiagonal golden, and
ScatterMaskedIndexSimplify ahead of the single-point DUS rewrite).

(cherry picked from commit e64e6b9)
(cherry picked from commit 5739b52)
wsmoses added a commit that referenced this pull request Sep 8, 2026
scatter_dus_guard / scatter_masked_index: with #2993's ScatterMaskedIndexSimplify and #3022's provable-bounds guard in
the same pipeline, a masked single-index scatter has its mask resolved into
the update and then stays a scatter at the live (unproven) index — neither
PR's golden alone described the pair.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
(cherry picked from commit b1882dd)
(cherry picked from commit 2459cf4)
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 30.49%. Comparing base (a1d5338) to head (e496b38).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2993      +/-   ##
==========================================
+ Coverage   29.44%   30.49%   +1.05%     
==========================================
  Files         238      238              
  Lines       48117    48576     +459     
==========================================
+ Hits        14166    14814     +648     
+ Misses      33951    33762     -189     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zErYp7upmqr4NHfhod9UD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants