Repository navigation
perf: push left filters through ASOF joins - #24801
Merged
Merged
Conversation
github-actions
Bot
added
sql
This was referenced Aug 30, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24801 +/- ##
=======================================
Coverage 82.41% 82.42%
=======================================
Files 1138 1139 +1
Lines 435299 435374 +75
Branches 435299 435374 +75
=======================================
+ Hits 358749 358838 +89
+ Misses 54840 54834 -6
+ Partials 21710 21702 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Sep 4, 2026
## Which issue does this PR close? - Part of #318. - Umbrella PR: #23738. - Depends on #23828 (merged). ## Rationale for this change This is the logical-planning layer of the ASOF JOIN stack. It defines the logical contract and planner behavior separately from the SQL frontend and serialization formats. #23828 is merged, so this PR's diff against `main` is the isolated logical layer. It no longer depends on the optional floating-point follow-up #24375. ## What changes are included in this PR? - Add `LogicalPlan::AsOfJoin`, `AsOfJoin`, and `AsOfMatch`. - Validate deterministic expressions, input ownership, supported match operators, equality-key types, and USING constraints. - Add `LogicalPlanBuilder` entry points and schema construction that preserves both qualified `USING` keys while exposing one unqualified wildcard key. - Integrate ASOF joins with tree transforms, display, type coercion, projection pruning, row bounds, and physical planning. - Plan logical ASOF joins to the broadcast-based `AsOfJoinExec` from #23828. - Fail closed at proto, SQL unparser, and Substrait boundaries until their owning stack layers add explicit support. - Defer ASOF-specific functional-dependency refinement to #24799 and filter pushdown to #24801 so each optimization can be reviewed independently. ## Are these changes tested? Yes: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-expr min_rows_of_joins --all-features` - `cargo test -p datafusion-substrait asof_join_fails_closed_until_substrait_has_an_extension --all-features` - The extended workspace test command from the contributor guide ## Are there any user-facing changes? This adds logical-plan and builder APIs for ASOF joins. SQL syntax, DataFrame APIs, and plan serialization are intentionally left to dependent stack PRs. Floating equality keys remain rejected by the merged physical operator unless the independent follow-up #24375 is also included. As with any new public `LogicalPlan` variant, downstream exhaustive matches must add an arm. The variant is appended so existing variants retain their `PartialOrd` ordering; maintainers should still treat the enum addition as a Rust source-compatibility break. This PR can be reviewed independently now that #23828 has merged. The optimization follow-ups #24799 and #24801 are not required by the core ASOF stack.
osipovartem
added a commit
to Embucket/datafusion
that referenced
this pull request
Sep 13, 2026
* feat: add ASOF join physical operator (apache#23828) ## Which issue does this PR close? - Part of apache#318. - Umbrella PR: apache#23738. - Follow-up for floating-point equality keys: apache#24375. ## Rationale for this change This is the first layer of the ASOF JOIN stack. It establishes a broadcast-based physical execution contract independently so later floating-point equality, logical-plan, SQL, DataFrame, and serialization changes can be reviewed as smaller follow-up PRs. The initial implementation deliberately favors the simpler broadcast design: the right input must fit in memory and each left partition scans the shared right-side batches. A repartitioned implementation can be evaluated separately without changing the ASOF semantics introduced here. Floating-point equality keys are rejected in this base layer because Arrow's required sort order distinguishes `-0.0` from `+0.0` while join equality does not. apache#24375 adds the required ordering normalization as an independently reviewable layer. ## What changes are included in this PR? - Add `AsOfJoinExec` for left-preserving, Snowflake-style ASOF semantics. - Coalesce and collect the ordered right input once, then share it across all left partitions. - Keep the left input partitioned so each partition can scan independently and preserve the left-side output partitioning. - Preserve merge state across input and output batch boundaries. - Reserve each retained Arrow buffer exactly once, including when right-side batches are zero-copy slices, and expose build, match, and output metrics. - Define output properties and statistics for the broadcast execution model. - Reject floating-point equality keys until apache#24375 supplies a sort/equality contract that handles signed zero correctly. - Add physical operator tests covering match directions, equality groups, batch boundaries, unmatched rows, invalid contracts, shared-buffer memory accounting, multi-partition broadcast execution, and float-key rejection. ## Are these changes tested? Yes: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-physical-plan joins::asof_join --all-features` - Extended workspace tests from the contributor guide - FFI integration tests ## Are there any user-facing changes? This adds a new physical operator API. The base operator deliberately rejects floating-point equality keys; apache#24375 adds full Float16, Float32, and Float64 support. SQL and DataFrame APIs are left to later dependent PRs. --------- Co-authored-by: Yongting You <2010youy01@gmail.com> * feat: add ASOF join logical semantics (apache#23829) - Part of apache#318. - Umbrella PR: apache#23738. - Depends on apache#23828 (merged). This is the logical-planning layer of the ASOF JOIN stack. It defines the logical contract and planner behavior separately from the SQL frontend and serialization formats. logical layer. It no longer depends on the optional floating-point follow-up - Add `LogicalPlan::AsOfJoin`, `AsOfJoin`, and `AsOfMatch`. - Validate deterministic expressions, input ownership, supported match operators, equality-key types, and USING constraints. - Add `LogicalPlanBuilder` entry points and schema construction that preserves both qualified `USING` keys while exposing one unqualified wildcard key. - Integrate ASOF joins with tree transforms, display, type coercion, projection pruning, row bounds, and physical planning. - Plan logical ASOF joins to the broadcast-based `AsOfJoinExec` from - Fail closed at proto, SQL unparser, and Substrait boundaries until their owning stack layers add explicit support. - Defer ASOF-specific functional-dependency refinement to apache#24799 and filter pushdown to apache#24801 so each optimization can be reviewed independently. Yes: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-expr min_rows_of_joins --all-features` - `cargo test -p datafusion-substrait asof_join_fails_closed_until_substrait_has_an_extension --all-features` - The extended workspace test command from the contributor guide This adds logical-plan and builder APIs for ASOF joins. SQL syntax, DataFrame APIs, and plan serialization are intentionally left to dependent stack PRs. Floating equality keys remain rejected by the merged physical operator unless the independent follow-up apache#24375 is also included. As with any new public `LogicalPlan` variant, downstream exhaustive matches must add an arm. The variant is appended so existing variants retain their `PartialOrd` ordering; maintainers should still treat the enum addition as a Rust source-compatibility break. This PR can be reviewed independently now that apache#23828 has merged. The optimization follow-ups apache#24799 and apache#24801 are not required by the core ASOF stack. * feat: support ASOF JOIN SQL (apache#23830) - Part of apache#318. - Umbrella PR: apache#23738. - Builds on apache#23829 and apache#23828, both merged. This is the SQL frontend layer of the ASOF JOIN stack. It adds syntax and unparsing on top of the merged physical and logical contracts. The PR now contains only the isolated SQL frontend diff. - Plan `ASOF JOIN ... MATCH_CONDITION (...)` with optional `ON` or `USING` equality keys. - Reject unsupported match shapes and non-equality `ON` predicates. - Unparse ASOF joins while preserving right-side candidate preselection and nested join scope. - Document the supported SQL syntax and semantics, including qualified `USING` keys, left-partitioned broadcast execution, the full-right memory requirement, repeated scans, and the absence of spill/repartitioned ASOF. - Add SQL integration and sqllogictest coverage for all four match directions, coercion, equality-free joins, USING, invalid contracts, EXPLAIN, boundedness, and optimized-plan round trips. - Verify the broadcast topology with a multi-partition left input: left partitioning is preserved while the right input is single-partitioned. Yes: - `cargo fmt --all` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion --test core_integration asof --all-features` - `cargo test -p datafusion-sqllogictest --test sqllogictests --all-features -- asof_join` - The extended workspace test command from the contributor guide Users can express Snowflake-style ASOF joins in SQL with `MATCH_CONDITION`, optional equality keys, and `<`, `<=`, `>`, or `>=` match directions. With `USING`, wildcard output exposes one unqualified key while both qualified input keys remain addressable. The user guide also documents the initial broadcast strategy and its memory/no-spill limitations. stack and does not depend on the optional floating-point follow-up apache#24375. --------- Co-authored-by: Xuanwo <github@xuanwo.io> Co-authored-by: Yongting You <2010youy01@gmail.com>
Xuanwo
force-pushed
the
xuanwo/asof-filter-pushdown
branch
from
September 20, 2026 14:09
3a5e377 to
3c05055
Compare
github-actions
Bot
removed
sql
Xuanwo
marked this pull request as ready for review
September 20, 2026 14:10
Xuanwo
force-pushed
the
xuanwo/asof-filter-pushdown
branch
from
September 20, 2026 15:11
3c05055 to
3c71cdb
Compare
Member
Author
|
Hi @2010YOUY01 and @jayzhan211, this ASOF filter pushdown follow-up is ready for review. Would you mind taking a look when you have time? Thanks! |
jayzhan211
reviewed
Sep 21, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Xuanwo! A few suggestions
jayzhan211
approved these changes
Sep 22, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Xuanwo , LGTM!
dwsmith1983
pushed a commit
to dwsmith1983/datafusion
that referenced
this pull request
Sep 26, 2026
## Which issue does this PR close? - Part of apache#318. - Follow-up to apache#24801. ## Rationale for this change When a filter selects one ASOF equality-key group on the left, rows in other right-side groups cannot match any surviving left row. The broadcast join still sorts and retains those right rows unless the same key filter is applied to the right input. ## Performance Q09 has 50M rows on each side across 500K equality groups. `WHERE l.key = 42` retains one group and produces 100 rows with both revisions. On an Apple M4 Max, using independent release builds and two 7-iteration runs per revision, the median elapsed time across all 14 iterations was: | Revision | Median elapsed | | --- | ---: | | `main` without this optimization (`92559afc0`) | 972.1 ms | | This change (`c7522fa37`) | 78.5 ms | This selective keyed workload is about **12.4x faster** (91.9% lower elapsed time). The benchmark file and command were the same for both binaries: ```shell target/release/benchmark_runner asof_join --query 9 --iterations 7 ``` ## What changes are included in this PR? - Mirror a pushed `left_key = literal` predicate to the corresponding right key when both join keys are direct columns of the same type. - Keep the original left filter and ASOF null-padding behavior unchanged. Other predicate shapes and coerced or expression-based keys are not inferred. - Add result and plan coverage in `asof_join.slt`, plus a keyed selective workload as ASOF benchmark Q09. ## What is the testing strategy for this PR? The SLT cases check the mirrored plan, matching and unmatched results, a right-side `IS NULL` filter, and a left disjunction that must not prune the right input. Q09 runs the user-visible query with 50M rows per side across 500K groups, selecting one group. ## Are there any user-facing changes? No API or result changes. Eligible ASOF joins can sort and retain fewer right-side rows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
An ASOF join emits each left row exactly once and does not change left-side
values. A deterministic predicate that references only the left input can
therefore run before matching. Right-only and mixed predicates must remain
above the join because unmatched right fields are NULL-padded, and volatile
predicates must keep their original evaluation point.
Pushing the safe subset reduces the rows entering the ASOF join and can expose
the predicate to further scan-level pushdown.
Performance
Q08 models a high-selectivity case with 10M ordered left rows, 100K ordered
right rows, and a left predicate that retains 1% of the input. It is included
in the ASOF benchmark suite.
On an Apple M4 Max, using a release build and two 7-iteration runs per
revision:
1cfc35579(then-currentmain)This workload is 6.17x faster (83.8% lower elapsed time) because the
join processes 100K left rows instead of 10M.
What changes are included in this PR?
asof_join.slt.Are these changes tested?
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo test --profile ci -p datafusion-optimizer --all-featurescargo test --profile ci --test sqllogictests -- asof_join.sltcargo run --profile ci --bin benchmark_runner -- asof_join --query 8 --iterations 1Are there any user-facing changes?
No result semantics or public APIs change. Eligible ASOF joins may execute with
fewer left-side rows after logical optimization.