Skip to content

fix: reject GROUPING masks wider than 64 bits - #26137

Open
alexandrefimov wants to merge 4 commits into
apache:mainfrom
alexandrefimov:codex/datafusion-grouping65-publication-20261008
Open

alexandrefimov wants to merge 4 commits into
apache:mainfrom
alexandrefimov:codex/datafusion-grouping65-publication-20261008

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #26135.

Rationale for this change

GROUPING over 65 distinct grouping columns can panic while the analyzer constructs the mask.

What changes are included in this PR?

Reject unrepresentable grouping-set masks after argument validation and before constructing the mask. Preserve the ordinary GROUP BY path where GROUPING returns zero.

What is the testing strategy for this PR?

Five core API tests cover high-bit, low-bit and all-argument cases, 63/64-key boundaries, ordinary 65-key GROUP BY and COUNT-only physical-capacity refusal. Both aggregate kernels are covered.

Focused tests, crate tests, extended workspace tests, formatting, all-target/all-feature Clippy and dev/rust_lint.sh passed.

Are there any user-facing changes?

Unsupported masks return NotImplemented during analysis.

Reject grouping-set layouts with more than 64 distinct keys before the
analyzer constructs a UInt64 GROUPING mask. Preserve argument validation
and the ordinary GROUP BY path where GROUPING evaluates to zero.

Add core API tests for unrepresentable masks, valid boundaries, ordinary
65-key grouping and COUNT-only physical-capacity refusals in both
aggregate kernels.
@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate labels Oct 8, 2026
@codecov-commenter

codecov-commenter commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.77%. Comparing base (06aa131) to head (6bc318b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26137      +/-   ##
==========================================
- Coverage   82.77%   82.77%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      450909   450913       +4     
  Branches   450909   450913       +4     
==========================================
+ Hits       373252   373255       +3     
+ Misses      54934    54930       -4     
- Partials    22723    22728       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@neilconway neilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking at this, @alexandrefimov !

Comment thread datafusion/core/tests/dataframe/mod.rs Outdated
Comment thread datafusion/optimizer/src/analyzer/resolve_grouping_function.rs Outdated
Comment thread datafusion/core/tests/dataframe/mod.rs Outdated
Move the grouping bitmap capacity check to Aggregate::try_new and cover
the SQL paths with logic tests instead of kernel-specific API tests.
@github-actions github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) and removed optimizer Optimizer rules core Core DataFusion crate labels Oct 8, 2026
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

The macOS runner lost connection. Could you rerun the failed job? I don’t have permission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject oversized GROUPING masks during analysis

3 participants