Repository navigation
Commit b73054a
authored
feat: allow scaling RangePartitioning (#24766)
## Which issue does this PR close?
- Closes #24712.
## Rationale for this change
Distributed and adaptive planners need to change a range-partitioned
plan's task count without losing its higher-resolution boundary sample.
The optimizer should use a compatible range layout where possible while
preserving the requested parallelism when sample capacity is
insufficient.
## What changes are included in this PR?
- Add validated sample-backed construction, sample/current/maximum
partition-count accessors, and `scale(usize) -> Option<Self>`. Zero and
above-capacity requests return `None`.
- Retain samples across scaling, projection, repartition, join rewrites,
protobuf, and FFI round trips. Keep structural equality for interleave
safety and use `has_same_layout` for effective-layout comparisons.
- Integrate scaling into `EnsureRequirements`, recheck key requirements
after singleton scaling, and fall back to key partitioning at the
requested parallelism when scaling is unsupported. During
co-partitioning, a source already providing the required layout remains
native; a newly inserted range exchange does not gain that preference.
- Let `RepartitionExec::repartitioned` scale supported ranges with fresh
runtime state and metrics, retaining ordering and batch-size settings.
Remove the optimizer's redundant outer `Option`.
- Deprecate unchecked `new` for 56.0.0, retain validated exact
construction and `split_points()`, and document sample sizing and
upgrade requirements.
- Preserve effective boundaries in protobuf field 2 for older readers,
and validate the added sample/count representation against those
boundaries.
## What is the testing strategy for this PR?
Coverage includes count limits, sample retention, key and sort-option
compatibility, structural versus effective-layout equality, interleave
safety, sample-preserving rewrites, protobuf/FFI round trips, and
aggregate/join results with NULLs.
The new exchange regression fails before the follow-up fix and passes
afterward. It initializes the original exchange, scales 2 -> 4 -> 1 ->
4, and checks exact row routing, ordering, retained samples, batch
configuration, and independent execution state and metrics. The 23 range
optimizer integration regressions also pass.
A separate join regression fails before the native-reference fix and
passes afterward. It checks both input orders and verifies that a
larger, newly scaled range exchange does not displace a source already
providing the required hash layout. The end-to-end range join test now
checks the resulting hash layout and exact output rows when scaling
requires exchanges on both sides.
Validation with the pinned Rust 1.98.1 toolchain:
- Formatting, all-target/all-feature Clippy with warnings denied, and
the complete `./dev/rust_lint.sh` suite pass.
- The extended workspace suite passes 12,022 Rust tests, with eight
ignored, and all 522 SQL logic-test files.
- All 155 all-feature FFI tests pass.
Performance caveat: six existing range benchmarks were compared with
base `925d7f8ffd` using separate Rust 1.98.1 release-nonlto builds and
alternating runs. The eight-partition integer routing case was 13.5%
slower; the other five differed by -1.1% to +2.2%. A separate
before/after check of that case found this follow-up within 0.3% of
pre-follow-up head `1d2d59e699`, with both about 18% slower than the
base in that run. This indicates an existing full-PR performance
concern, not a measured slowdown introduced by this follow-up. Its cause
remains unverified; these local results do not establish performance
parity with main.
## Are there any user-facing changes?
Yes. `RangePartitioning` gains sample-backed construction and scaling
APIs. `scale` returns `Option<Self>` rather than a public error type.
Unchecked `new` is deprecated; validated exact construction remains
supported. Unsupported optimizer scaling falls back to key partitioning
rather than retaining lower parallelism.
`FFI_RangePartitioning` changes layout and the generated
`PhysicalRangePartitioning` protobuf struct gains fields. These API/ABI
changes require the `api change` label. This targets main, not a patch
release. FFI consumers must rebuild against the compatible release.
Existing protobuf payloads remain readable.
---------
Signed-off-by: goutamadwant <workwithgoutam@gmail.com>1 parent f50c17d commit b73054a
20 files changed
Lines changed: 1614 additions & 135 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- ffi
- src
- physical_expr
- tests
- tests
- physical-expr/src
- physical-optimizer/src/ensure_requirements
- physical-plan/src
- joins
- repartition
- proto-models
- proto
- src/generated
- proto
- src/physical_plan
- tests/cases/plans
- sqllogictest
- src/test_context
- test_files
- docs/source/library-user-guide/upgrading
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
600 | 600 | | |
601 | 601 | | |
602 | 602 | | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
603 | 609 | | |
604 | 610 | | |
605 | 611 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
65 | | - | |
66 | | - | |
| 66 | + | |
| 67 | + | |
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
| |||
83 | 84 | | |
84 | 85 | | |
85 | 86 | | |
86 | | - | |
| 87 | + | |
87 | 88 | | |
| 89 | + | |
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
| |||
105 | 107 | | |
106 | 108 | | |
107 | 109 | | |
108 | | - | |
109 | | - | |
| 110 | + | |
| 111 | + | |
110 | 112 | | |
111 | 113 | | |
112 | 114 | | |
| |||
126 | 128 | | |
127 | 129 | | |
128 | 130 | | |
129 | | - | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
130 | 136 | | |
131 | 137 | | |
132 | 138 | | |
| |||
210 | 216 | | |
211 | 217 | | |
212 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
213 | 250 | | |
214 | 251 | | |
215 | 252 | | |
216 | | - | |
| 253 | + | |
217 | 254 | | |
| 255 | + | |
218 | 256 | | |
219 | 257 | | |
220 | 258 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
290 | 290 | | |
291 | 291 | | |
292 | 292 | | |
293 | | - | |
| 293 | + | |
294 | 294 | | |
295 | 295 | | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
296 | 299 | | |
297 | | - | |
| 300 | + | |
298 | 301 | | |
299 | 302 | | |
300 | 303 | | |
| |||
314 | 317 | | |
315 | 318 | | |
316 | 319 | | |
317 | | - | |
318 | 320 | | |
319 | 321 | | |
320 | 322 | | |
| |||
351 | 353 | | |
352 | 354 | | |
353 | 355 | | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
354 | 360 | | |
355 | 361 | | |
356 | 362 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | | - | |
33 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
| |||
228 | 231 | | |
229 | 232 | | |
230 | 233 | | |
231 | | - | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
232 | 250 | | |
233 | 251 | | |
234 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
145 | 169 | | |
146 | 170 | | |
147 | 171 | | |
| |||
0 commit comments