Repository navigation

[bench] combine apache#24086 (read-ahead) + apache#25752 (optional filters) - #82

Closed

adriangb wants to merge 65 commits into

mainfrom

combine-24086-25752

Commits on Sep 28, 2026
- fix: do not install a Parquet row selection when the page index prunes nothing
  
  adriangb
  and
  claude
  committed
  446765a
- perf: collapse partitioned hash join InList dynamic filters into one IN list
  
  adriangb
  and
  claude
  committed
  7607cb8
- perf: dedupe collapsed hash join IN lists and add combined bounds
  
  adriangb
  and
  claude
  committed
  d2a3eb3
- perf: push partitioned hash join dynamic filter bounds and membership as separate filters
  
  adriangb
  and
  claude
  committed
  205c2e7
- bench: add ordered-subset hash join dynamic filter benchmark
  
  adriangb
  and
  claude
  committed
  bbbd3a3
- fix: rename test to pass the typos check
  
  adriangb
  and
  claude
  committed
  9219137
- feat(parquet-datasource): always accept pushable filters, run rejected conjuncts post-scan
  
  adriangb
  and
  claude
  committed
  e74402b
- Update tests added on main for always-accepted parquet filters
  
  adriangb
  and
  claude
  committed
  588e894
- fix: stop the Parquet push decoder stream after the final flush
  
  adriangb
  and
  claude
  committed
  7af48ce
- refactor(physical-plan): share the round-robin row check of EnforceDistribution
  
  adriangb
  and
  claude
  committed
  c5ce4b2
- fix(datasource): keep a filter above a scan that cannot give the target partitions
  
  adriangb
  and
  claude
  committed
  bbb6193
- feat: add OptionalFilterPhysicalExpr to mark filters not needed for correctness
  
  adriangb
  and
  claude
  committed
  b003a46
- feat: keep optional filters pruning-only in the Parquet post-scan path
  
  adriangb
  and
  claude
  committed
  8f894aa
- feat: add filter_stats (FilterCost, Clock) for filter measurements
  
  adriangb
  and
  claude
  committed
  d0bd6e9
- feat: adaptive conjunct reordering in FilterExec
  
  adriangb
  and
  claude
  committed
  5c81d84
- feat: add OptionalFilterGate to pause optional filters that cost more than they save
  
  adriangb
  and
  claude
  committed
  0bbc09e
- feat(gate): add a measured overhead for each evaluated row to the cost
  
  adriangb
  and
  claude
  committed
  81cafbd
- feat(gate): share pauses between the gates of one plan site
  
  adriangb
  and
  claude
  committed
  3eecaed
- feat(gate): decide only on windows of at least MIN_OBSERVED_ROWS rows
  
  adriangb
  and
  claude
  committed
  68d5038
- feat(gate): use the measured work of the filter producer as the saving
  
  adriangb
  and
  claude
  committed
  7b412f4
- feat: add optional_filter_mode config
  
  adriangb
  and
  claude
  committed
  b9c2579
- feat: let the Parquet scan skip optional filters that are not worth their cost
  
  adriangb
  and
  claude
  committed
  d2b303e
- test: end-to-end tests for optional filters in the Parquet post-scan path
  
  adriangb
  and
  claude
  committed
  9bd9ec1
- feat: let FilterExec skip optional filters that are not worth their cost
  
  adriangb
  and
  claude
  committed
  3802295
- test(filter): size the FilterExec optional filter tests for gate windows
  
  adriangb
  and
  claude
  committed
  06fa789
- feat: mark hash join, TopK and aggregate dynamic filters as optional
  
  adriangb
  and
  claude
  committed
  da18a81
- test: update plan snapshots for optional dynamic filters
  
  adriangb
  and
  claude
  committed
  586f5e9
- feat(hash-join): measure the work that a dynamic filter saves for each removed row
  
  adriangb
  and
  claude
  committed
  9c3922b
- fix(hash-join): measure only the work of a probe row without a match
  
  adriangb
  and
  claude
  committed
  76e4b9d
- feat(parquet): adaptive placement of scan filter conjuncts
  
  adriangb
  and
  claude
  committed
  c5ed30c
- test: sqllogictests for adaptive filter placement
  
  adriangb
  and
  claude
  committed
  19dc188
- feat(parquet): carry row selections across placement rebuilds
  
  adriangb
  and
  claude
  committed
  8a166f4
- feat(parquet): measure skippable rows on the stage input
  
  adriangb
  and
  claude
  committed
  6d81b2f
- feat(parquet): start filters post-scan and add a row filter stage cost
  
  adriangb
  and
  claude
  committed
  ad398d8
- fix(parquet): hand coalesced rows downstream at row group boundaries
  
  adriangb
  and
  claude
  committed
  647af41
- fix(parquet): count the actual rows in optional_filter_rows_skipped
  
  adriangb
  and
  claude
  committed
  e34984d
- fix(parquet): count only skippable rows in the saving of an optional row filter
  
  adriangb
  and
  claude
  committed
  3d5fbb4
- feat(parquet): share the pauses of optional filter gates in a scan
  
  adriangb
  and
  claude
  committed
  a455e18
- feat(parquet): place and order required and optional conjuncts with one model
  
  adriangb
  and
  claude
  committed
  56a99da
- feat(parquet): add hysteresis to the placement changes of a file
  
  adriangb
  and
  claude
  committed
  e7e9928
- feat(parquet): measure unmeasured conjuncts first and forget old filters
  
  adriangb
  and
  claude
  committed
  5791de6
- test(parquet): gate windows of MIN_OBSERVED_ROWS rows in row filter tests
  
  adriangb
  and
  claude
  committed
  72f6691
- refactor(parquet): use the shared MIN_OBSERVED_ROWS in the placement model
  
  adriangb
  and
  claude
  committed
  6276115
- feat(parquet): let measured evidence decide the placement, drop the statistics prior
  
  adriangb
  and
  claude
  committed
  0b80d81
- feat(parquet): order row filter predicates by measured rows removed per nanosecond
  
  adriangb
  and
  claude
  committed
  79f9869
- refactor(parquet): share the ranking and the hysteresis with the row filter order
  
  adriangb
  and
  claude
  committed
  9616f5c
- feat(parquet): add the measured post-scan copy to the benefit of a row filter
  
  adriangb
  and
  claude
  committed
  cd05bb2
- test(parquet): size the optional filter integration tests for gate windows
  
  adriangb
  and
  claude
  committed
  1060d95
- feat(parquet): order the post-scan conjuncts by their measurements at each batch
  
  adriangb
  and
  claude
  committed
  6a3a7da
- fix(parquet): fix rustdoc links of the adaptive filter placement
  
  adriangb
  and
  claude
  committed
  22b411f
- fix(parquet): compact the post-scan batch with the pre-selection threshold of AND
  
  adriangb
  and
  claude
  committed
  4fac55a
- test: optional dynamic filters in runtime row group pruning tests
  
  adriangb
  and
  claude
  committed
  3318ef1
- fix(parquet): no scan equivalences from a pruning-only predicate
  
  adriangb
  and
  claude
  committed
  c32df68
- fix(parquet): no scan equivalences from optional conjuncts
  
  adriangb
  and
  claude
  committed
  45dcc5e
- fix(parquet): no scan equivalences from optional conjuncts
  
  adriangb
  and
  claude
  committed
  a7b6bc5
- docs(parquet): optional conjuncts of the placement are not exact
  
  adriangb
  and
  claude
  committed
  2cd2a65
- test: the scan applies the filter of the wrapping multiplication test
  
  adriangb
  and
  claude
  committed
  bd93482
- feat: make adaptive optional filters and adaptive filter placement the default
  
  adriangb
  and
  claude
  committed
  b88a5b5
- test: run runtime row group pruning tests with the new defaults
  
  adriangb
  and
  claude
  committed
  3e5f53c
- test: optional filters start in the post-scan filter in sqllogictests
  
  adriangb
  and
  claude
  committed
  f1036e8
Commits on Sep 29, 2026
- chore: pin arrow-rs with batch-granular push decoding and scan_plan
  
  adriangb
  and
  claude
  committed
  f679435
- POC: batch-granular Parquet scans with the push decoder and scan_plan read-ahead
  
  adriangb
  and
  claude
  committed
  754d844
- remove read_ahead_conditional; read-ahead always fetches conditional ranges
  
  adriangb
  and
  claude
  committed
  667c4dc
- account Parquet read-ahead in the MemoryPool
  
  adriangb
  and
  claude
  committed
  02429ca
- Merge PR 24086 (read-ahead) into PR 25752 (optional filters)
  
  adriangb
  and
  claude
  committed
  07caaa3

Commits

Commits on Sep 28, 2026

Commits on Sep 29, 2026