fix: backport upstream fixes for wrong results from filter pushdown, dynamic filters, simplification and planning by lukekim · Pull Request #235 · spiceai/datafusion

lukekim · 2026-09-26T22:30:27Z

Which issue does this PR close?

Backports upstream fixes for queries that return wrong rows. None of them is in DataFusion 54.1, and Spice reaches each one with its defaults: dynamic filter pushdown is on for joins, TopK and aggregates, and every Spice session sets parquet.pushdown_filters = true, so a Parquet scan applies a pushed filter row by row, as a Cayenne (Vortex) scan does for every predicate it accepts.

Rationale for this change

Each fix returns rows that differ from SQL's answer, with no error.

Filter pushdown and dynamic filters:

Upstream	Wrong result it fixes
apache#24817	`MIN(a), MAX(a), MAX(b), MIN(c + 1)`: the aggregate's dynamic filter was built from the plain-column aggregates alone and pruned the rows holding `MIN(c + 1)`
apache#25259	A join or `TopK` dynamic filter pushed below an operator with two same-named columns (nested joins, a filter or projection, a `GROUP BY`) was resolved by name onto the wrong column: the join lost its only match, or `ORDER BY p.amount LIMIT 1` returned the wrong row
apache#23104	`x NOT IN (subquery)` returned rows although the subquery holds a NULL: the filter pushed into the subquery's scan dropped its NULLs
apache#23173	`x NOT IN (subquery)` kept a NULL `x` when the pushed filter emptied the subquery's scan
apache#22965, apache#23106	A null-equal join (`IS NOT DISTINCT FROM`, `INTERSECT`) lost its `NULL = NULL` match. apache#22965 disables the filter for such joins; apache#23106 re-enables it with `OR key IS NULL`
apache#22926	A filter holding both a grouping-column and an aggregate-output predicate lost the latter (reached when that filter survives to the physical plan)
apache#24045	A filter above an anti join was taken as handled when only the non-output side accepted it (same reach as apache#22926)

apache#22926, apache#24045 and apache#24428 (a FilterRemapper fast path) are the prerequisites apache#25259 is written against. They are applied first, so apache#25259 applies as upstream wrote it.

Planning, simplification and execution:

Upstream	Wrong result it fixes
apache#22810	With `prefer_hash_join = false` and more than one partition, `NOT IN (subquery)` was planned as a sort-merge join, which has no null-aware mode, and a NULL in the subquery stopped excluding rows
apache#23684	The `TopK` aggregation dropped groups whose `MIN`/`MAX` is NULL
apache#24247	`log(a, 1)`, `log(a, a)`, `power(a, 0)` and their kin were folded to constants, wrong where `a` is NULL
apache#24248	`(a XOR b) XOR a` was folded to `b` and `a XOR a` to `0`, wrong where `a` is NULL
apache#24380	`col ~ '.*'` was folded to `true`, wrong where `col` is NULL
apache#24686	Merging nested projections that redefine a column used the wrong layer: six `i + 1` layers added three
apache#24958	A sort pushed below a `GlobalLimitExec` ignored its `skip`: `ORDER BY` over `LIMIT 4 OFFSET 3` returned one row
apache#24997	`COUNT(a, c ORDER BY b)` counted only `a`, and a grouped `COUNT(a ORDER BY b)` panicked
apache#25348	`3 NOT IN (subquery)` returned rows although the subquery holds a NULL
apache#25227	`MIN`/`MAX` over a cast column was answered from uncast Parquet statistics: `MAX(CAST(a AS INT))` over `'1'`, `'100'`, `'2'` answered `2`
apache#24516	A scalar subquery that returns no rows is NULL, but it was typed with its projected field's nullability, so `(SELECT 1 WHERE FALSE) IS NULL` was folded to `false`

apache#23429 (an IN subquery is nullable when its subquery's column is) is the prerequisite apache#24516 is written against. On this branch its own effect does not reach a result: an IN subquery that is not a top-level WHERE conjunct is evaluated by a mark join, which is not null-aware here.

A Spice patch for correlated NOT IN. Backported alone, apache#25348 left correlated NOT IN worse than before on 54.1. Its null-aware anti join takes one key and checks the subquery's NULLs over every build row. With a correlation, that has four consequences:

A constant NOT IN with an equality correlation applies the NULL rules to the correlation key, so a NULL correlation key drops rows although the subquery holds no NULL. Before fix: NOT IN (subquery) with a constant value ignores NULLs in the subquery apache/datafusion#25348 this shape answered correctly on the data below.
A subquery NULL that the correlation excludes still removes every outer row.
A column NOT IN with an equality correlation fails to plan ("null_aware anti join only supports single column join key").
A constant NOT IN with a non-equality correlation fails to plan ("requires equi-join keys").

Upstream fixes these in the null-aware join executor (apache#25339, apache#25560), on top of the null-aware mark joins of apache#21585, which this branch does not have. Instead, 76c676a7e plans a correlated NOT IN in a WHERE clause as the NOT EXISTS it equals there: a plain anti join on x IS NULL OR y IS NULL OR x = y and the correlation. An uncorrelated NOT IN keeps the null-aware hash join. With non-nullable columns the disjunction simplifies to x = y, which stays an equi-join key.

When the decorrelation drops a subquery filter as a duplicate of the IN equality (… WHERE y = x), the subquery keeps only values equal to x. Then x = y and the correlation decide on their own. The null-aware plan got this wrong when no other correlation was left, because it treated the subquery as uncorrelated.

The NOT EXISTS form is not used for a scalar aggregate that the decorrelation groups by the correlation, which loses the row the aggregate returns over no input. It is also not used for a volatile x, which the filter would evaluate more than once. Both keep the plan they had.

It is a Spice patch, not a backport: drop it on a re-cut onto a release that has apache#25339 and apache#25560. Against SQLite, on 81 queries covering 9 NOT IN shapes over 9 datasets that put NULLs in the outer value, the subquery's values and the correlation keys:

	wrong rows	planning error
`spiceai-54` (`11624fb82`)	20	18
this branch with apache#25348 alone (`5ac6b8edb`)	19	27
this branch	0	0

The 81 queries are datafusion/sqllogictest/test_files/not_in_correlated.slt.

Not backported. apache#24932 and apache#25234 build on larger upstream changes this branch does not have (EnsureRequirements, a time ± interval result-type change). A NOT IN under OR or IS NULL is evaluated by a mark join, which is not null-aware here (apache#25560, on apache#21585). Their wrong results remain on the fork.

What changes are included in this PR?

Twenty-seven commits. Each is an upstream commit cherry-picked with -x, except the correlated NOT IN patch above and five test-only adaptations to 54.1, which say so:

b0288090b fix: aggregate dynamic filtering with unsupported expressions (#24817)
67535bb11 fix: preserve aggregate filter pushdown order (#22926)
4ea30ba97 fix: prevent incorrect results when pushing filters through anti joins (#24045)
d78f26192 perf: skip building FilterRemapper when there are no parent filters (#24428)
5c24978bb fix: map pushed-down filter columns by position instead of by name (#25259)
6205fdac7 fix: keep null-aware anti-join NULLs in the pushed dynamic filter (#23104)
c23a61b22 fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (#23173)
85400b666 test: drop an aggregates test import that 54.1 does not use
0444d4da9 fix: Disable join dynamic filters for null-equal joins (#22965)
30538e660 fix: re-enable null-equal join dynamic filters with an IS NULL predicate (#23106)
67c2051d0 fix: NOT IN with NULL subquery returns wrong results under SortMergeJoin (#22810)
f6222e333 fix: TopK aggregation drops groups whose MIN/MAX value is NULL (#23684)
31012b52c fix: preserve NULL semantics in `log` and `power` simplification (#24247)
bb82ff026 fix: preserve NULL semantics in bitwise xor simplification (#24248)
53e60cd7b fix: preserve NULL semantics when simplifying col ~ '.*' (#24380)
cbc04ddb3 fix: preserve nested projection expressions (#24686)
5a0f29501 fix: sort pushed below `GlobalLimitExec` ignores `skip`, returning too few rows (#24958)
42d388899 fix: prevent panic and incorrect results for COUNT with ORDER BY (#24997)
cc926a0c1 fix: `NOT IN (subquery)` with a constant value ignores NULLs in the subquery (#25348)
794e2826d fix: only propagate cast statistics through safe conversions (#25227)
fdfaec8d0 test: drop the single-pass collapse assertion from merge_deep_projection_chain_in_one_pass
70834cde1 test: use a NOT NULL table for the non-nullable log/power bases plan
5ac6b8edb test: expect the NOT IN plan without the null_aware display marker
76c676a7e fix: plan a correlated `NOT IN (subquery)` as the `NOT EXISTS` it equals in a filter
1ef0eee67 test: render the #22810 NOT IN plan with the fork's HashJoinExec accumulator display
624470fcb fix: Fix nullability of logical `InSubquery` expression  (#23429)
d1815c2a3 fix: account for empty scalar subqueries in nullability (#24516)

Every conflict resolution is recorded in its commit's message. They are mechanical: test scaffolding and helpers 54.1 does not have, and .slt expectations rendered in 54.1's display (no dynamic_rg_pruning field, no null_aware marker, the fork's accumulator= text, the byte counts 54.1's Parquet writer produces). Predicates, rows and pruning metrics are as upstream wrote them. The only code taken from outside a fix is a 7-line test helper from apache#21585, named in the apache#25348 commit.

Merging with Rebase and merge keeps each backport as its own commit with its (cherry picked from commit …) line, which is what the next re-cut audit reads; a squash folds them into one.

Are these changes tested?

Each fix's upstream regression query returned the wrong rows on the commit before it and the right rows after, with the same command. On the final head:

datafusion-physical-plan  filter_pushdown 11, aggregates 109, joins::hash_join 389   all pass
datafusion-optimizer 709, datafusion-physical-optimizer 62, datafusion-physical-expr 1553 (2 ignored),
datafusion-expr 233,
datafusion-functions 284, datafusion-functions-aggregate 124                          all pass
cargo check -p datafusion -p datafusion-substrait -p datafusion-proto                 0 warnings, 0 errors
cargo clippy (the six crates above, --all-targets; datafusion --lib)                  0 warnings

The full sqllogictest suite, compared against the base with the same test files, differs only in the queries these fixes change and the ones they add. The seven core-integration filter_pushdown failures and the .slt failures that remain are the fork's pre-existing accumulator= display drift; their results do not change with this branch. On the final head, the same 27 .slt files fail as on spiceai-54 (11624fb82), at the same queries.

On the Spice side, a guard for every fix runs its upstream regression query in a session built from Spice's default configuration: crates/cayenne/tests/datafusion_dynamic_filter_backports_test.rs for the filter-pushdown fixes (over Parquet, and over Cayenne tables where the shape is reachable) and crates/runtime-datafusion/src/fork_backport_guards.rs for the rest. All 21 fail on the current pin (11624fb82d) and pass on this branch; the three for the commits after 5ac6b8edb also fail at 5ac6b8edb. They land with the pin bump in spiceai/spiceai#14430, together with the docs/dev/fork_patches.md rows.

Are there any user-facing changes?

Queries that returned wrong rows now return SQL's answer, and correlated NOT IN shapes that failed to plan now run. A plan for a mixed MIN/MAX aggregate no longer carries a dynamic filter, and a null-equal join's pushed filter gains OR key IS NULL.

…#24817) ## Which issue does this PR close?  - Closes apache#24816. ## Rationale for this change  Aggregate dynamic filtering currently ignores unsupported expressions while still generating a shared scan predicate from supported aggregates. This can prune rows required by the unsupported aggregate and produce incorrect results. ## What changes are included in this PR?  - Disable aggregate dynamic filtering when any aggregate expression is unsupported. - Document that aggregate dynamic filtering requires every aggregate expression to be supported. - Update the existing sqllogictest to verify that no dynamic filter is produced for mixed supported and unsupported expressions. ## What is the testing strategy for this PR?  Yes. Updated the existing sqllogictest case in `push_down_filter_regression.slt`. ## Are there any user-facing changes?  Yes. This fixes potentially incorrect query results. No API changes. (cherry picked from commit b3cb365) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/aggregates/mod.rs: the base spells the cols_for_dynamic_filter check as debug_assert!(a == b); applied only the upstream rename (supported_accumulators_info -> accumulator_dyn_filter_info) to that line. - datafusion/sqllogictest/test_files/push_down_filter_regression.slt: the base carries an older version of the mixed-expressions section; replaced it with the upstream section verbatim. The upstream aggregate_stream.rs hunks apply to aggregates/no_grouping.rs, its name on 54.1.

## Which issue does this PR close? - Closes apache#22925. ## Rationale for this change `FilterPushdown` maps each child pushdown result back to its original parent filter by position. `AggregateExec::gather_filters_for_pushdown` previously split parent filters into safe and unsafe buckets and then concatenated the results, which changed their order. For example, in `cnt@2 = 1 AND b@1 = bar`, the grouping-column predicate on `b` can cross the aggregate, while the predicate on aggregate output `cnt` must remain above it. Reordering the returned results could make the optimizer associate the pushed-down `b` result with the `cnt` predicate and incorrectly remove the latter. Aggregate pushdown also needs to account for empty-input semantics. A global aggregate, or a grouping-sets aggregate containing `()`, can emit a row even when its input is empty. Moving any parent predicate below such an aggregate — including a column-free predicate such as `false` — can therefore change the result. ## What changes are included in this PR? - Preserve parent-filter order by constructing each child description once with `ChildFilterDescription::from_child_with_allowed_indices`. - Allow only grouping-output columns that are present in every grouping set; aggregate-result columns remain above the aggregate. - Mark all parent filters unsupported for global aggregates and grouping sets containing an empty grouping set, while retaining aggregate-generated dynamic filters. - Validate that every child returns one parent-filter result per input filter before positional remapping. - Add physical optimizer and SQL regression coverage for mixed filter order, global-aggregate name collisions, constant predicates, and grouping sets. ## Are these changes tested? Yes: - `cargo test -p datafusion --test core_integration physical_optimizer::filter_pushdown` - `cargo fmt --all -- --check` - `cargo check -p datafusion-physical-plan -p datafusion-physical-optimizer` - `git diff --check upstream/main` - Full GitHub CI, including Rust tests, clippy, and sqllogictests ## Are there any user-facing changes? There are no public API changes. Filter pushdown now preserves mixed aggregate predicates correctly and avoids moving predicates across aggregates when doing so could change empty-input results. (cherry picked from commit 2f25454)

apache#24045) ## Which issue does this PR close? - Closes apache#24002. ## Rationale for this change Pushing filters to an anti join's non-output side can produce incorrect results. ## What changes are included in this PR? Restrict anti-join filter pushdown to the output side while preserving two-sided join-key pushdown for semi joins. ## Are these changes tested? Yes, with updated unit and SQL logic tests. ## Are there any user-facing changes? no API changes but some downstream expected plans may change (cherry picked from commit a3f0f93) Conflict resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt: the expected physical plan for the NOT IN + TopK query keeps the 54.1 display (no null_aware marker on HashJoinExec, no dynamic_rg_pruning field). With this change the TopK filter no longer reaches the non-output side, so the right-side scan drops from two dynamic filters to one; the one that remains is the hash join's own probe-side filter, which upstream no longer pushes for a null-aware anti join with a nullable build key (5975a2d, apache#23173, not on 54.1). Query results are unchanged from upstream.

…pache#24428) ChildFilterDescription::from_child (and from_child_with_allowed_indices) built a FilterRemapper before checking whether parent_filters was empty. FilterRemapper::new indexes every column of the child's schema into a HashMap, and with no filters to remap that index is never used: remap_filters returns an empty description regardless of what the remapper contains. Each plan node produces one ChildFilterDescription per child, so a union pays this cost once per child. On a 250,000-column scan a sampling profile put FilterRemapper::new at 794 of 20,373 samples, almost entirely HashMap insertion and hashing. Return ChildFilterDescription::empty() directly when parent_filters is empty, before the remapper is built. ## Which issue does this PR close?  Do I need an issue for this change, it feels like a really minor fix? Co-authored-by: kosiew <kosiew@gmail.com> (cherry picked from commit 6cc3442)

…pache#25259) ## Which issue does this PR close? - Closes apache#25244. - Closes apache#25262. - Closes apache#25263. - Closes apache#25264. - Closes apache#21246. - Closes apache#25296. ## Rationale for this change Enabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` can silently drop matching rows when the probe side of a join contains several columns with the same name, for example `a.id` and `b.id` from a nested join. Physical filter pushdown resolved a pushed filter's columns in the child schema by name, so a predicate on the second `id` column was rewritten to the first `id` column, which holds a different value. The same name-based mapping also affects TopK dynamic filters under the default configuration (apache#25296). For a nested join of orders and payments that both expose `amount`, `ORDER BY p.amount LIMIT 1` can push the payment threshold onto `orders.amount`. Later matching orders are incorrectly pruned, returning `(100, 30)` instead of `(300, 10)`. Every operator that forwards parent filters had the same weakness, in slightly different forms: - `HashJoinExec` computed which output columns belong to each side by position, but then resolved the child column by name. - `FilterExec`, `SortExec`, `RepartitionExec`, `CoalesceBatchesExec`, `UnionExec` and similar nodes used the generic name lookup even though their output positions equal their input positions. - `FilterExec` with an embedded projection resolved unconsumed parent filters back into input coordinates by name. - `ProjectionExec` looked up output aliases by name to find the expression to substitute, so two outputs aliased `id` both mapped to the first. - `AggregateExec` restricted pushdown to grouping positions but resolved the input column by name. The join-filter regression queries return one row with dynamic filtering disabled and zero rows with it enabled across these shapes. The `FilterExec`, `ProjectionExec` and `AggregateExec` cases were found while auditing the remaining name-based paths for apache#25244; they share the root cause, so this PR fixes them together. The TopK case in apache#25296 is fixed by the same positional mapping. ## What changes are included in this PR? Built-in operators now remap filter columns by position. The deprecated public API retains its historical name-based resolution for compatibility. - `FilterRemapper` uses a `ColumnMapping` enum: `Identity` preserves positions and checks that names match; `Explicit` uses a caller-supplied parent-output to child-input mapping and allows names to differ. - `ChildFilterDescription::from_child` now maps by position. New `ChildFilterDescription::from_child_with_column_mapping` takes an explicit `HashMap<usize, usize>`. - `HashJoinExec` builds the explicit mapping from its `column_indices` and output projection. For semi joins, output join keys on the emitted side are mapped to the paired key on the other side, which also supports differently named keys. - `FilterExec` maps through its embedded projection in both pushdown phases and when folding unconsumed parent filters back into its predicate. - `ProjectionExec` substitutes the expression at each output position instead of looking the alias up in the output schema. - `AggregateExec` maps each grouping output position to the input column that grouping expression reads; non-column grouping expressions are not forwarded, as before. - `ChildFilterDescription::from_child_with_allowed_indices` is kept as a deprecated wrapper that preserves name-based resolution to the first matching child field. It translates allowed parent column references into an explicit positional mapping; new callers should supply positions directly to avoid ambiguous duplicate names. ## What is the testing strategy for this PR? - `dynamic_filter_pushdown_config.slt` gains four regression queries over two small Parquet tables, run once with join dynamic filtering disabled and once enabled, asserting identical rows: nested joins with `RepartitionExec` between them (the query from apache#25244), a `FilterExec` with an embedded projection, a `ProjectionExec` with duplicate aliases, and an `AggregateExec` grouping on same-named columns. All four return zero rows on `main` with dynamic filtering enabled. - `dynamic_filter_pushdown_config.slt` also covers apache#25296 using nested joins of orders, payments and customers, with one row per batch and one row per orders Parquet row group. The query returns `(300, 10)` with TopK dynamic filtering disabled, enabled, and enabled together with Parquet filter pushdown. On `main` at `85d4cbb0a9`, both enabled cases reproduce the incorrect `(100, 30)` result; all three pass on this branch. - `datafusion/core/tests/physical_optimizer/filter_pushdown.rs` gains focused tests that call `gather_filters_for_pushdown` directly on `RepartitionExec`, `HashJoinExec` (duplicate child columns with and without projection, both semi join directions with differently named keys, and a semi join whose key is not a plain column), `FilterExec` with a projection in both phases, `ProjectionExec` with duplicate aliases, and `AggregateExec` with reordered same-named grouping columns. Further tests cover the deprecated `from_child_with_allowed_indices` wrapper preserving the first name match, accepting allowed parent indices outside the child schema, and rejecting unresolvable names, the identity mapping rejecting a column whose name differs from the child field at that position, and `FilterExec` rejecting an unconsumed parent filter outside its projection. Passed locally on the updated implementation: - `cargo fmt --all` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - `cargo test --profile ci -p datafusion --test core_integration physical_optimizer::filter_pushdown` — 70 tests passed. - `cargo test --profile ci --test sqllogictests -- dynamic_filter_pushdown_config.slt` — passed. - The isolated apache#25296 regression fails on `main` in both TopK-enabled configurations with the expected wrong-result mismatch. ## Are there any user-facing changes? Queries that push join or TopK dynamic filters through operators with duplicate column names now return the correct rows, including the default-configuration TopK query in apache#25296. API change in `datafusion-physical-plan`: `ChildFilterDescription::from_child_with_allowed_indices` is deprecated in favour of `from_child` and the new `from_child_with_column_mapping`; the deprecated function preserves its previous name-based behavior. Callers should migrate to explicit positions because duplicate names make name resolution ambiguous. `ChildFilterDescription::from_child` also resolves by position, which only affects callers that used it on a node whose output positions differ from its child's. (cherry picked from commit b376290) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/projection.rs: applied the upstream removal of collect_reverse_alias and of the PhysicalColumnRewriter and DataFusionError imports it used. Kept the 54.1 neighbours the upstream context does not have (the CardinalityEffect-only import, with_new_children_and_same_properties) and did not add upstream-only context (replace_children_if_necessary, plan_err, compute_overrides_metadata, overrides_metadata). - docs/source/library-user-guide/upgrading/56.0.0.md: dropped. It documents a 56.0.0 API change and does not exist on the 54.x branch. - datafusion/core/tests/physical_optimizer/filter_pushdown.rs: the new HashJoinExec tests rely on the module-level JoinType, HashJoinExec and PartitionMode imports that the upstream file has; added those two import entries (the 54.1 tests import them inside each test function instead).

…ache#23104) ## Which issue does this close? Closes apache#23103. ## Rationale for this change A hash join pushes a build-side dynamic filter (`key IN build_keys`) down to the probe scan. For a null-aware anti join (`NOT IN`), that filter drops the probe's NULL rows. But `NOT IN` three-valued logic needs a probe-side NULL to collapse the whole result to zero rows. With the NULL filtered away at the scan, before the join's null-aware check runs, the join returns rows that shouldn't be there. ## What changes are included in this PR? `SharedBuildAccumulator::build_filter` now ORs `probe_key IS NULL` into the pushed predicate when the join is `null_aware`. Non-NULL probe rows still get filtered, so the optimization stays. `HashJoinExec`'s `null_aware` validation already guarantees a single probe key. ## Are these changes tested? Yes. Added a parquet-backed case to `null_aware_anti_join.slt`. The existing cases use in-memory `VALUES`, whose scans never apply the pushed filter, so they passed despite the bug. The new one sets `parquet.pushdown_filters = true` so the filter runs row-level. Without the fix it returns `1, 3`; with it, zero rows. ## Are there any user-facing changes? A `NOT IN` over a NULL-bearing inner now returns zero rows instead of leaking rows, when join dynamic filter pushdown and row-level scan filtering are both on. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: rjhallsted <rjhallsted@gmail.com> (cherry picked from commit 8d680db) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: both hunks are in test scaffolding. Upstream adds null_aware: false to the make_partitioned_accumulator_for_test and completed_partitions_for_test helpers, which 54.1 does not have, so they are not added. In the tests module helper, 54.1 builds probe_schema locally where upstream calls test_probe_schema(); kept the 54.1 form and added null_aware: false. The accumulator changes themselves applied as upstream wrote them; the fork's CollectLeftAccumulator seam only computes bounds, and the pushed filter is still published only at the two update sites this wraps.

…llable build key (apache#23173) ## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown. (cherry picked from commit 5975a2d) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/exec.rs: added only this commit's null-aware gate to allow_join_dynamic_filter_pushdown and its two unit tests. The surrounding upstream context (the NullEqualsNull gate and its test from baa497d, apache#22965, and the swap-inputs and range-partitioning tests) is not on 54.1 and is not added. - datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt: kept the 54.1 plan display (no null_aware, no dynamic_rg_pruning) and removed the hash join's own dynamic filter from the probe scan of both NOT IN plans. For the NOT IN + TopK plan that leaves the right-side scan with no predicate: the TopK filter was already removed there by a3f0f93 (apache#24045), which is upstream's state once both changes are in. - datafusion/sqllogictest/test_files/explain_tree.slt: dropped. The change edits a NOT IN record that 127731b (apache#22913) added and 54.1 does not have. - datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: added only this commit's regression block, not the null-equal join block from baa497d (apache#22965) that precedes it upstream.

The backport of b376290 (apache#25259) moved `use std::collections::HashSet` from the module scope of aggregates/mod.rs into its test module, where upstream's tests use it. None of the 54.1 tests in that module do, so the import was unused and `cargo test -p datafusion-physical-plan` warned about it.

## Which issue does this PR close? - Closes #apache#22964 ## Rationale for this change We presently allow dynamic filter pushdown to be applied to null-equal hash joins. This might result in pushing a predicate down into the probe-side plan, where the predicate will not be evaluated with the null-equal semantics that are required. Longer-term, we might consider supporting this case with the correct semantics (e.g., generate a predicate with `OR IS NULL ...`), but for now disabling pushdown for null-equal joins seems much more practical. ## What changes are included in this PR? * Disable hash join dynamic filter pushdown for null-equal joins * Add SLT test with end-to-end repro * Add unit test ## Are these changes tested? Yes. ## Are there any user-facing changes? No. (cherry picked from commit baa497d) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/exec.rs: this branch already carries 5975a2d (apache#23173), which upstream merged later and which adds its own gate and tests at the same places. Placed this commit's gate and test ahead of those, the order they have upstream. - datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: same; the null-equal block goes ahead of the apache#23173 block, as upstream has it.

…ate (apache#23106) ## Which issue does this close? Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the proper null-equal semantics. ## Rationale for this change apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti joins; this re-enables the null-equal case the same way. ## What changes are included in this PR? - Revert the null-equal `return false` in `allow_join_dynamic_filter_pushdown`. - Generalize the shared probe-NULL helper to cover both null-aware (single-key) and null-equal (multi-key) joins: OR `key IS NULL` for every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity. ## Are these changes tested? Yes. apache#22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and `preserve_probe_nulls` unit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening). ## Are there any user-facing changes? Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results. (cherry picked from commit 0d5f9b1) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/exec.rs: renamed the null-equal test as upstream does; its upstream neighbour (test_swap_inputs_rejects_dynamic_filter) is not on 54.1. - datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: test scaffolding only. Skipped the upstream-only helpers (including the reported() helper this commit edits); added null_equality to the 54.1 test helper; imported Column in the test module, which upstream's module already does and the new tests need. - datafusion/physical-plan/src/joins/hash_join/stream.rs: upstream's test module there is built on a BuildReportHandle API that 54.1 does not have, so its one-line change is not applied. 54.1 reports a dropped partition through report_canceled_partition (CanceledUnknown), which this change already treats as possibly holding a NULL key. - datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: the five new EXPLAIN / EXPLAIN ANALYZE expectations follow the 54.1 display: no dynamic_rg_pruning field, and the scan_efficiency_ratio byte counts of the files 54.1's parquet writer produces. Predicates, row counts and pruning metrics are as upstream wrote them.

…oin (apache#22810) ## Problem `NOT IN (subquery)` is a null-aware anti join: when the subquery yields a NULL the predicate is never TRUE, so the query must return zero rows. With `prefer_hash_join = false` and multiple partitions, the planner routed the null-aware anti join to `SortMergeJoinExec`, which is not null-aware, so it returned wrong results. HashJoin (the default) was already correct. ## Proof ```sql set datafusion.optimizer.prefer_hash_join = false; create table t1(x int) as values (1); create table t2(y int) as values (NULL); select x from t1 where x not in (select y from t2); ``` Expected 0 rows (the subquery contains a NULL). Before this change it returned `1`. With `prefer_hash_join = true` it correctly returned 0 rows. `EXPLAIN` showed the wrong config selecting `SortMergeJoinExec: join_type=LeftAnti`. ## Solution The planner already requires null-aware joins to use the CollectLeft HashJoin, and the HashJoin branch guards on `!null_aware`. The SortMergeJoin branch was missing the same guard, so this adds `&& !*null_aware` to it. Null-aware anti joins now fall through to the CollectLeft HashJoin regardless of `prefer_hash_join`. `SortMergeJoinExec` has no `null_aware` parameter and cannot honor these semantics. Added a regression test in `subquery.slt` (under `prefer_hash_join = false`) covering both a null-containing subquery (zero rows) and a null-free subquery (normal anti join). All 61 SortMergeJoin unit tests pass. (cherry picked from commit 82f1b36)

…e#23684) ## Which issue does this PR close? - Closes apache#23440 - Closes apache#22190 ## Rationale for this change When `TopKAggregation` pushes a `LIMIT` into a MIN/MAX aggregate, a group whose aggregate inputs are all NULL can disappear instead of being returned with a NULL aggregate value. The stream previously skipped NULL aggregate inputs without registering the group. This is correct for an individual MIN/MAX input, but not for a group whose inputs are all NULL. There is also an important correctness boundary: nullable MIN/MAX with `NULLS FIRST` is not monotonic for a bounded aggregation. A group can start at NULL, later become non-NULL, and thereby move to a worse rank. Keeping only `limit` NULL candidates can therefore discard a group that belongs in the final result. ## What changes are included? - Track up to `limit` all-NULL candidates alongside valued TopK groups and emit them for the parent sort to rank and truncate. - Correctly convert a tracked NULL group when its first value arrives, or unregister it when that value cannot enter the valued TopK. - Skip TopK pushdown for nullable MIN/MAX with `NULLS FIRST`; regular aggregation is used for exact results. TopK remains enabled for `NULLS LAST`, non-nullable MIN/MAX inputs, and GROUP BY-only/DISTINCT queries. - Replace the single reusable hash-table slot with a free-slot stack. A NULL-to-value conversion can free both the NULL registration and an evicted valued group, so retaining only one slot caused unbounded backing-store growth under repeated conversions. - Select NULL-aware insertion once per batch, keeping NULL bookkeeping off the common no-NULL per-row hot path. - Add regression coverage for all-NULL groups, mixed NULL/value batches, bounded NULL candidate backfill, evicted groups, hash-table slot reuse, and optimizer plan selection. ## Are these changes tested? Yes. The following passed on the final commit: - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` - TopK physical-plan unit tests (33 tests) - `aggregates_topk.slt` and affected `group_by.slt` tests - The repository's extended workspace command with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`, including all 495 sqllogic files and extended/fuzz suites ## Performance The `topk_aggregate` 10-million-row time-series benchmark exposed an initial ~5.5% regression from checking NULL state on every row. Moving NULL handling to a batch-selected slow path removed the measurable regression. Final 30-sample 95% intervals on the same machine and settings: - `main`: 25.630–26.436 ms - this PR: 25.151–27.061 ms ## Are there any user-facing changes? Queries that previously dropped all-NULL groups under `ORDER BY <min/max> ... LIMIT` now return correct SQL results. Nullable MIN/MAX queries using `NULLS FIRST` may use regular aggregation rather than the bounded TopK optimization to guarantee correctness. There are no API or configuration changes. (cherry picked from commit f8b9ed8) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/sqllogictest/test_files/aggregates_topk.slt: three expected-plan blocks conflicted, and a fourth changed record needed the same treatment. Upstream's plans put the TopK SortExec below the ProjectionExec, a sort pushdown that 54.1 does not do. The four EXPLAIN expectations follow the 54.1 plan, which keeps the SortExec above the ProjectionExec. The AggregateExec lines, which carry this commit's decision (lim=[N] kept for NULLS LAST and removed for NULLS FIRST), are as upstream wrote them. - The upstream grouped_topk_stream.rs change applies to aggregates/topk_stream.rs, its name on 54.1.

…che#24247) ## Which issue does this PR close?  - Part of apache#24246. ## Rationale for this change  `log` and `power` simplifications removed a nullable base expression, which could incorrectly produce a non-NULL result when the base was NULL. For example: ```sql SELECT log(a, 1.0), power(a, 0.0) FROM (VALUES (NULL::DOUBLE)) AS t(a); ``` These expressions should both return NULL, but simplification could replace them with 0.0 and 1.0. ## What changes are included in this PR? only apply these simplifications when the removed base expression is non-nullable.  ## Are these changes tested?  Yes. Added sqllogictests covering results and plans. ## Are there any user-facing changes? Yes. log and power expressions with nullable bases now correctly preserve NULLs.  (cherry picked from commit c08832d) Conflict resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/functions/src/math/power.rs: context only. Upstream computes return_type ahead of the null-propagation branch, which 54.1 does not; added this commit's base_nullable binding in the 54.1 position. Both `&& !base_nullable` guards apply to the 54.1 arms unchanged.

) ## Which issue does this PR close?  - Part of apache#24246 . ## Rationale for this change Bitwise XOR simplifications cancelled repeated nullable operands, which could incorrectly produce a non-NULL result when the operand was NULL. For example: ```sql SELECT i XOR i, (i XOR 7) XOR i, i XOR (7 XOR i) FROM (VALUES (NULL::INT)) AS t(i); ``` These expressions should all return NULL, but simplification could replace them with 0, 7, and 7.  ## What changes are included in this PR? Only cancel repeated XOR operands when the removed operand is non-nullable.  ## Are these changes tested?  Yes. Added sqllogictests covering results. ## Are there any user-facing changes? Yes. Bitwise XOR expressions with repeated nullable operands now correctly preserve NULLs.  (cherry picked from commit 50cdbde)

## Which issue does this PR close? - Closes apache#24379. ## Rationale for this change `simplify_regex_expr` rewrites `col ~ '.*'` to `col IS NOT NULL`. For a NULL input that returns `false`, but `NULL ~ '.*'` is `NULL` under three-valued logic — so the rewrite produces wrong results in a projection context: ```sql SELECT s, s ~ '.*' FROM (VALUES (CAST(NULL AS VARCHAR)), ('x')) t(s); -- NULL row currently returns `false`; it should be NULL ``` The `!~` (`RegexNotMatch`) branch of the same rule is already NULL-aware (`col IS NULL AND NULL`); only the `~` branch dropped the NULL. ## What changes are included in this PR? - Rewrite `col ~ '.*'` to `col IS NOT NULL OR NULL` — `true` for a non-NULL string, `NULL` for a NULL input. - In a WHERE filter both FALSE and NULL reject the row, so filter *results* are unchanged; only the plan text and projection-context values differ. Existing filter-plan expectations in `simplify_expr.slt` and the `test_simplify_regex_special_cases` unit test are updated accordingly, and a projection regression test is added. ## Are these changes tested? Yes. - New projection regression test in `simplify_expr.slt` asserting `col ~ '.*'` returns `true`/`true`/`NULL` for `'foo'`/`''`/`NULL`. - Updated the two filter-context plan expectations (logical + physical) that previously encoded the `IS NOT NULL` rewrite. - `simplify_expr.slt`, the `regexp/*` SLTs, and the optimizer simplify unit tests all pass. ## Are there any user-facing changes? `col ~ '.*'` in a projection now returns `NULL` for a NULL input instead of `false`, matching SQL semantics. No API changes. (cherry picked from commit 16b08db)

## Which issue does this PR close? - No issue has been filed. ## Rationale for this change Anonymous nested projections that reuse the same output name can return wrong results. For example, each `i + 1 AS i` layer must be evaluated independently, but the projection optimizer could drop one layer when two consecutive projection expression vectors were structurally equal. Structural equality does not imply that a projection is safe to elide: repeated computations such as `i + 1 AS i` have the same expression shape but must still be evaluated twice. ## What changes are included in this PR? - Remove the structural-equality fast path that directly elided one of two consecutive projections. - Keep the existing iterative whole-chain merge, so deep projection chains still collapse within one optimizer rule invocation. - Continue using the normal projection rewrite path, which composes repeated expressions and preserves aliases and field metadata. - Add focused regressions for repeated non-idempotent projections, one-pass collapse of a 12-level chain, and metadata-bearing aliases. - Add an execution-level SQLLogicTest with six anonymous `i + 1 AS i` layers under both `max_passes = 1` and the default optimizer configuration. ## What is the testing strategy for this PR? The focused unit tests verify that: - two structurally equal `i + 1 AS i` projections retain both additions; - a 12-level chain preserves all 12 additions and collapses to one `Projection` with `max_passes = 1`; - a metadata-bearing `Alias(Column)` is merged without losing field metadata. The SQLLogicTest executes a six-level anonymous projection chain against a temporary table with both one optimizer pass and the default pass count. Verified with: ```text cargo fmt --all --check cargo test -p datafusion-optimizer optimize_projections # 58 passed; 0 failed cargo test -p datafusion-optimizer --test optimizer_integration # 26 passed; 0 failed cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- projection.slt # 1/1 files completed; 0 failures cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings # passed git diff --check # passed ``` ## Are there any user-facing changes? Yes. Deep anonymous nested projections that reuse an output name now preserve every projection expression and return the correct result. There are no public API or configuration changes. --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit 7a1f468)

…o few rows (apache#24958) ## Which issue does this PR close? - Closes #. ## Rationale for this change `ORDER BY` applied on top of a `LIMIT ... OFFSET` subquery returns too few rows. ```sql SELECT * FROM (SELECT a FROM t1 LIMIT 4 OFFSET 3) ORDER BY a; ``` `EnforceSorting` pushes the sort below the `GlobalLimitExec` and converts it into a TopK. The TopK's fetch was seeded from `GlobalLimitExec::fetch()`, which is the limit's *output* row count and ignores `skip`. So the TopK kept only 4 rows, the limit then skipped 3 of them, and the query returned a single row instead of 4. The same seeding happens in two places in `sort_pushdown.rs`: when a root's direct child is a `GlobalLimitExec` (`assign_initial_requirements`) and when the pushdown descends through one (`pushdown_sorts_helper`). Both had the bug. ## What changes are included in this PR? - Add an `input_fetch` helper in `sort_pushdown.rs` that returns `skip + fetch` for `GlobalLimitExec` and plain `fetch()` for every other operator. - Use it at both sites where the pushed-down fetch is derived from the plan node, so the TopK below a `GlobalLimitExec` retains enough rows for the limit to skip and still return `fetch` rows. ## What is the testing strategy for this PR? - `limit.slt`: new `EXPLAIN` + result test for `LIMIT 4 OFFSET 3` under `ORDER BY`, showing `TopK(fetch=7)` below `GlobalLimitExec: skip=3, fetch=4` and the correct 4 rows. - `ensure_requirements.rs`: two unit tests, one per code path (`test_sort_pushed_below_limit_with_skip_keeps_skip_rows` and `test_parent_ordering_over_limit_with_skip_keeps_skip_rows`), asserting the pushed sort has `fetch=15` for `skip=5, fetch=10`. ## Are there any user-facing changes? Queries with a sort above `LIMIT ... OFFSET` now return the correct number of rows. No API changes. (cherry picked from commit 16ace4f) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-optimizer/src/enforce_sorting/sort_pushdown.rs (upstream moved it to ensure_requirements/enforce_sorting/): the two lines this commit changes (current_fetch now comes from input_fetch, and the child fetch is current_fetch) are applied to the 54.1 loop. The distribution requirement code around them upstream is from fb1c0f3 (apache#21976), which 54.1 does not have. Added the module-level GlobalLimitExec import that input_fetch needs; upstream's module had it from 408527a (apache#24932), which is not on this branch. - datafusion/core/tests/physical_optimizer/ensure_requirements.rs: dropped. It tests the EnsureRequirements rule from apache#21976, which 54.1 does not have. This commit's own input_fetch unit tests in sort_pushdown.rs are kept and pass.

…che#24997) ## Which issue does this PR close? - Closes apache#25055. ## Rationale for this change `COUNT` does not depend on input order, but it inherited the default `AggregateOrderSensitivity::HardRequirement`. As detailed in apache#25055, this could pass ordering keys to count accumulators as additional arguments, causing incorrect results or a panic, and could introduce unnecessary sorting. ## What changes are included in this PR? Override `Count::order_sensitivity` to return `AggregateOrderSensitivity::Insensitive`. This makes aggregate ordering keys ineffective for physical planning: they are excluded from accumulator inputs and no `SortExec` is required solely for `COUNT`'s `ORDER BY`. ## What is the testing strategy for this PR? Regression tests in `aggregate.slt` cover grouped, multi-argument, and non-grouped counts. The data includes both a null ordering key, which must not affect the count, and a null counted argument, which must still be excluded. A bare global `COUNT(a ORDER BY b)` over the in-memory test table is folded by `AggregateStatistics` to `num_rows - null_count(a)`, so `CountAccumulator` never runs. The test uses `a + 0`, which preserves `a`'s nullness but prevents that fold, ensuring the regression exercises the accumulator. An `EXPLAIN` assertion verifies that the count does not require a sort. ## Are there any user-facing changes? `COUNT(... ORDER BY ...)` returns the correct count without panicking. Nulls in counted arguments retain their usual behavior; nulls in ordering keys no longer incorrectly exclude rows. (cherry picked from commit 224cc56) Conflict resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/sqllogictest/test_files/aggregate.slt: upstream appends this commit's COUNT ... ORDER BY block after other blocks that are not on 54.1, including the nested-aggregate block. Added only this commit's 43 lines, at the end of the 54.1 file.

…ubquery (apache#25348) ## Which issue does this PR close? - Closes apache#25340. A follow-up issue covers the remaining `NOT IN` problems. See **Follow-up work** at the end. ## Rationale for this change `3 NOT IN (1, NULL)` is UNKNOWN. A `WHERE` clause must remove the row. DataFusion kept every row. There was no error and no warning. ```sql CREATE TABLE t1(id INT) AS VALUES (1), (2); CREATE TABLE t2(id INT) AS VALUES (1), (NULL); SELECT id FROM t1 WHERE 3 NOT IN (SELECT id FROM t2) ORDER BY id; ``` | | result | |---|---| | DataFusion (before) | `1, 2` ❌ | | DataFusion (after) | *(no rows)* ✅ | | DuckDB 1.5.2 | *(no rows)* | | PostgreSQL 17.11 | *(no rows)* | The subquery gives `{1, NULL}`. The value `3` is not `NULL`, but `3` is also not known to be absent. Therefore the answer is UNKNOWN for every row of `t1`. The same expression in a `SELECT` list was already correct. Only the `WHERE` clause was wrong. ### Why it happened The value `3` holds no column. Therefore it could not become a join key. It stayed as a join filter. Two problems followed. ``` BEFORE AFTER WHERE 3 NOT IN (SELECT id FROM t2) WHERE 3 NOT IN (SELECT id FROM t2) │ │ ▼ ▼ anti join, no key anti join, key = (3, t2.id) filter: 3 = t2.id no filter │ │ ├─ (1) the filter moves into the ├─ (1) nothing to move │ subquery: WHERE t2.id = 3 │ │ the NULL row disappears │ │ │ └─ (2) no key, so the join becomes a └─ (2) the join has a key, so it nested loop join, which cannot stays a hash join, which do null-aware work. The flag is does null-aware work dropped without an error correctly │ │ ▼ ▼ 1, 2 ❌ (no rows) ✅ ``` ## What changes are included in this PR? Three changes. **1. Make the constant a join key** (`datafusion/optimizer/src/decorrelate_predicate_subquery.rs`) Add the constant to the outer side as a column. The comparison then becomes a true equality of two columns, so the existing null-aware hash join does the work. ``` Before: LeftAnti Join: Filter: Int64(3) = __correlated_sq_1.id null_aware → NestedLoopJoinExec (the null_aware flag is lost) After: LeftAnti Join: __correlated_sq_1_value = __correlated_sq_1.id null_aware Projection: t1.id, Int64(3) AS __correlated_sq_1_value → HashJoinExec ... null_aware ``` This applies only to uncorrelated subqueries. A correlated subquery needs a second key, and a null-aware anti join accepts only one key. **2. Keep the filter out of the subquery** (`datafusion/optimizer/src/push_down_filter.rs`) Do not move a predicate into the subquery side of a null-aware join. The NULLs must reach the join. `infer_join_predicates` has the same rule already. **3. Report an error instead of a wrong answer** (`datafusion/core/src/physical_planner.rs`) Only a hash join can do null-aware work, and it needs a key. If a null-aware join has no key, report an error. Do not build a nested loop join that gives wrong results without a warning. ## What is the testing strategy for this PR? New `sqllogictest` cases in `datafusion/sqllogictest/test_files/null_aware_anti_join.slt` and `null_aware_mark_join.slt`. They cover: - the four queries of the issue, and its controls; - a subquery with no NULL, which must keep all rows; - the `OR` and `IS NULL` forms, which use a mark join; - a user column with the same name as the new column, which must not be ambiguous; - the one shape that is not supported, which must report an error. New unit tests in `decorrelate_predicate_subquery.rs` and `push_down_filter.rs` hold the new plans. Results: | check | result | |---|---| | full `sqllogictest` suite | pass, and **no plan changes** anywhere else | | `datafusion-optimizer` unit tests | 853 pass | | extended workspace suite | 10,784 tests pass, 0 fail, 66 crates | | `cargo fmt --all` | clean | | `cargo clippy` | clean for the changed crates | The extended workspace figure comes from an earlier run of the same commit on a slightly older base. The full `sqllogictest` suite and the optimizer tests were re-run after the rebase onto current `main`. ## Are there any user-facing changes? Yes. Two. **1. `<constant> NOT IN (<subquery>)` in a `WHERE` clause now gives correct results.** This is the fix. **2. One shape now reports an error.** A constant with a correlation that is not an equality: ```sql SELECT id FROM t1 WHERE 3 NOT IN (SELECT id FROM t2 WHERE t2.g > t1.g); ``` ``` Error during planning: null_aware LeftAnti join requires equi-join keys, but the join has none ``` This query gave wrong results before. No correct result is lost. An error is better than a wrong answer that a user cannot see. There are no API changes. ## Follow-up work This PR makes every **uncorrelated** `NOT IN` correct. **Correlated** `NOT IN` still has problems. They come from the join operator, not from the code this PR changes. ``` NOT IN (subquery) │ ├── no outer column ──────────────▶ CORRECT (this PR) │ └── reads an outer column ├── equality condition ───────▶ WRONG or ERROR (follow-up, parts 1 and 2) └── other condition ──────────▶ WRONG (follow-up, part 3) ``` A test matrix of 18 query shapes gives these totals: | | wrong or error | |---|---| | before this PR | 13 of 18 | | after this PR | 10 of 18 | The 10 remaining shapes are all correlated. They fall into three parts: 1. A null-aware anti join accepts only one key, but a correlated `NOT IN` needs two. 2. A constant does not become a key when the subquery is correlated. 3. A null-aware join looks for NULLs only in the key, not in a leftover filter. Part 3 must come first, as an error. A prototype showed that part 2 alone turns a clear error into a silent wrong answer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VCeMPyJNgAiFpGXaz5CxF3 Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit edc936f) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/optimizer/src/decorrelate_predicate_subquery.rs: kept the 54.1 create_col_from_scalar_expr(right.deref(), alias) call and added this commit's lines around it. The new unit tests call the nullable_scalar_mark_scan test helper, which upstream added in bed9dcd (apache#21585, null-aware mark joins); added that 7-line helper verbatim. - datafusion/optimizer/src/push_down_filter.rs: moved the on_lr_is_preserved binding up with its null-aware override, as this commit does, and kept the 54.1 `if !on_filter.is_empty()` loop that uses it. Added only this commit's unit test; the neighbouring upstream test is from 0fcf628 (apache#23901), which is not on 54.1. - The three new unit tests' inline snapshots drop the ` null_aware` marker from the Join line: the 54.1 logical Join display does not print it (that came with 127731b, apache#22913). The plans are otherwise as upstream wrote them. - datafusion/sqllogictest/test_files/null_aware_anti_join.slt: added only this commit's block; the upstream block before it is from 0fcf628 (apache#23901). - datafusion/sqllogictest/test_files/null_aware_mark_join.slt: dropped. It tests null-aware mark joins from bed9dcd (apache#21585), which 54.1 does not have. So a constant IN / NOT IN that decorrelates to a mark join (inside an OR, or under IS NULL) is not fixed here. - Behaviour on 54.1: the new planner check makes a null-aware join without an equi-join key fail with "null_aware ... join requires equi-join keys" instead of running as a nested-loop join that ignores null-awareness. That is the shape a correlated `<constant> NOT IN` with a non-equality correlation produces on 54.1; upstream made it executable later.

…25227) ## Which issue does this PR close? - Closes apache#25221. ## Rationale for this change `MIN`/`MAX` over a cast can return incorrect results when aggregate optimization replaces the scan with converted column statistics. Casting the original endpoints is only valid if they remain extrema in the target domain. For example, a Parquet string column containing `('1', '100', '2')` has string extrema `'1'` and `'2'`. Previously, `MIN(CAST(a AS INT)), MAX(CAST(a AS INT))` could return `1, 2` from those statistics instead of the correct `1, 100`. The same issue affects `BIGINT`. ## What changes are included in this PR? - Restrict CAST statistics propagation to conversions proven safe for the source type or its exact integer bounds. Unsupported or unsafe conversions return unknown column statistics, so aggregates evaluate the data. - Preserve the existing same-type passthrough and supported lossless widening conversions. - Retain integer narrowing and signed/unsigned conversions when both original bounds are exact and non-null, and both converted endpoints are non-null. Successful endpoint conversions establish that the entire integer range fits in the target type; overflow errors or null endpoints do not qualify. - Extend `CastExpr::check_bigger_cast` to recognize `Int32 ↔ Date32`, and clarify that its contract includes lossless, strictly order-preserving conversions that preserve nulls. Arrow reinterprets the same `i32` values as days since the epoch, so these conversions preserve both statistics and ordering properties. This also retains the statistics-based `MIN`/`MAX` optimization for ClickBench's `UInt16 → Int32 → Date32` projection. ## What is the testing strategy for this PR? - Add projection unit tests for non-monotonic string-to-integer casts and integer narrowing with safe, overflowing, and inexact bounds. Removing the statistics guard makes the new string-cast regression test fail. - Add end-to-end Parquet regression cases in `parquet_statistics.slt` for `INT` and `BIGINT`, both expecting `1, 100`. - Add a bidirectional `Int32 ↔ Date32` test covering `i32::MIN`, `i32::MAX`, negative values, zero, NULL, and strict ordering properties. - Run the related projection and cast unit tests, plus the ClickBench and Parquet statistics SLT files. The existing ClickBench plan expectation passes unchanged, confirming that its aggregate still folds to constants. Local formatting, Clippy, license, spelling, workflow, and documentation checks also pass. ## Are there any user-facing changes? Affected casts now produce correct `MIN`/`MAX` results. Conversions whose safety is not established may require a scan instead of using statistics-based aggregate optimization. Supported safe integer conversions and `Int32 ↔ Date32` retain that optimization. (cherry picked from commit ccfe704) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-expr/src/projection.rs: added this commit's extrema check (min_value/max_value/preserves_values) and is_within_extrema to the 54.1 project_column_statistics_through_expr. Upstream's same-type early return in that function (d5bd10d, apache#24094) and its column_statistics_at helper are not on 54.1 and are not added. check_bigger_cast accepts identical types, so same-type casts still keep their extrema. - datafusion/physical-expr/src/expressions/cast.rs: the check_bigger_cast extension (Int32 <-> Date32, Int64 <-> Date64) and its doc comments apply as upstream wrote them. Not applied: the comment reword in cast_expr_properties, which is on strictly_order_preserving code 54.1 does not have, and the unit test test_integer_date_cast_preserves_values_and_ordering, which asserts that same property. - On 54.1, check_bigger_cast also gates substitute_cast_ordering in equivalence/properties, so the four added casts can now also replace a sort key there. They reinterpret the same integer and keep order and NULLs, which is the property upstream relies on in this commit.

…ion_chain_in_one_pass The backport of apache#24686 (cbc04dd) added `merge_deep_projection_chain_in_one_pass`, which optimizes a chain of twelve `i + 1 AS i` projections with `max_passes = 1` and asserts that 1. all twelve `Int32(1)` additions survive, which is what apache#24686 fixes, and 2. the chain collapses into a single `Projection`. The second assertion checks the whole-chain merge from upstream apache#22389 (0da8961), which this branch does not have; apache#24686 keeps that merge rather than adding it. On this branch the test fails on the second assertion only (`left: 1`, `right: 6`), after the first one has passed. The first assertion is kept. With the apache#24686 change to `merge_consecutive_projections_one_level` reverted, it fails with `left: 12`, `right: 7`, so the test still fails without the fix and passes with it.

The backport of apache#24247 (31012b5) added a `math.slt` plan check, "Non-nullable bases still use the existing simplifications", that reads its bases from `(VALUES (2.0::double, 3.0::double)) AS t(a, b)`. Upstream expects those columns to be non-nullable because of apache#22089 (e27b6c6, which infers VALUES nullability from the literals). This branch does not have apache#22089: `DESCRIBE` on a view over that VALUES list reports `YES` for both columns, so the fix correctly leaves the expressions unsimplified and the check fails. The check now reads the same values from a table whose columns are declared `NOT NULL`, which keeps what it verifies: a non-nullable base is still simplified. It passes with and without the apache#24247 change to `log.rs` and `power.rs`, as the upstream check does. The nullable-base query and plan checks before it are unchanged, and they fail without that change.

The backport of apache#22810 (67c2051) added a `subquery.slt` plan check whose expected `LeftAnti Join` and `HashJoinExec` lines end in ` null_aware`. That marker is printed by upstream apache#22913 (127731b), which this branch does not have, so the plan never shows it here. This is the same change made to the unit-test plans in the apache#25348 backport. The join itself is unchanged: with `prefer_hash_join = false`, the check still expects a CollectLeft `HashJoinExec`, not a `SortMergeJoinExec`. Its `HashJoinExec` line still lacks the `accumulator=` field that this fork prints on every `HashJoinExec`; no plan check in the sqllogictest files expects that field.

Copilot

Copilot review overview

🔵 Needs a closer look

The large cross-cutting backport changes multiple correctness-critical optimizer and execution paths and warrants final human review.

Review effort: Balanced
Findings: None

What changed in this PR

Backports upstream DataFusion correctness fixes for filter pushdown, dynamic filters, planning, aggregation, and expression simplification.

Changes:

Corrects null-aware/null-equal joins and position-based filter remapping.
Fixes TopK, sort/limit, projection merging, ordered COUNT, and cast statistics.
Adds extensive regression coverage for previously incorrect query results.

File	Description
`datafusion/sqllogictest/test_files/subquery.slt`	Tests null-aware join planning.
`datafusion/sqllogictest/test_files/simplify_expr.slt`	Tests nullable regex simplification.
`datafusion/sqllogictest/test_files/scalar.slt`	Tests nullable XOR expressions.
`datafusion/sqllogictest/test_files/push_down_filter_regression.slt`	Tests aggregate filter pushdown.
`datafusion/sqllogictest/test_files/push_down_filter_parquet.slt`	Tests join dynamic filters over Parquet.
`datafusion/sqllogictest/test_files/projection.slt`	Tests nested projection preservation.
`datafusion/sqllogictest/test_files/parquet_statistics.slt`	Tests statistics through casts.
`datafusion/sqllogictest/test_files/null_aware_anti_join.slt`	Tests `NOT IN` NULL semantics.
`datafusion/sqllogictest/test_files/math.slt`	Tests nullable log/power simplification.
`datafusion/sqllogictest/test_files/limit.slt`	Tests sorting across offset limits.
`datafusion/sqllogictest/test_files/group_by.slt`	Updates nullable TopK plans.
`datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt`	Tests positional dynamic-filter mapping.
`datafusion/sqllogictest/test_files/aggregates_topk.slt`	Tests all-NULL TopK groups.
`datafusion/sqllogictest/test_files/aggregate.slt`	Tests ordered `COUNT`.
`datafusion/physical-plan/src/projection.rs`	Remaps filters by output position.
`datafusion/physical-plan/src/joins/hash_join/stream.rs`	Reports build-key NULL presence.
`datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs`	Preserves probe NULLs in filters.
`datafusion/physical-plan/src/joins/hash_join/exec.rs`	Corrects join filter routing.
`datafusion/physical-plan/src/filter.rs`	Handles projected filter coordinates.
`datafusion/physical-plan/src/filter_pushdown.rs`	Introduces positional filter mappings.
`datafusion/physical-plan/src/execution_plan.rs`	Documents filter ordering contract.
`datafusion/physical-plan/src/aggregates/topk/priority_map.rs`	Tracks all-NULL TopK groups.
`datafusion/physical-plan/src/aggregates/topk/hash_table.rs`	Adds NULL-group storage and reuse.
`datafusion/physical-plan/src/aggregates/topk_stream.rs`	Processes nullable aggregate groups.
`datafusion/physical-plan/src/aggregates/no_grouping.rs`	Updates dynamic-filter metadata usage.
`datafusion/physical-plan/src/aggregates/mod.rs`	Tightens aggregate filter safety.
`datafusion/physical-optimizer/src/topk_aggregation.rs`	Disables unsafe NULLS FIRST TopK.
`datafusion/physical-optimizer/src/filter_pushdown.rs`	Validates parent-filter result ordering.
`datafusion/physical-optimizer/src/enforce_sorting/sort_pushdown.rs`	Accounts for limit offsets.
`datafusion/physical-expr/src/projection.rs`	Restricts cast-statistics propagation.
`datafusion/physical-expr/src/expressions/cast.rs`	Extends safe cast classification.
`datafusion/optimizer/src/simplify_expressions/simplify_exprs.rs`	Updates regex plan expectation.
`datafusion/optimizer/src/simplify_expressions/regex.rs`	Preserves regex NULL semantics.
`datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs`	Prevents nullable XOR cancellation.
`datafusion/optimizer/src/push_down_filter.rs`	Protects null-aware subquery inputs.
`datafusion/optimizer/src/optimize_projections/mod.rs`	Preserves repeated projection expressions.
`datafusion/optimizer/src/decorrelate_predicate_subquery.rs`	Creates keys for constant `NOT IN`.
`datafusion/functions/src/math/power.rs`	Guards nullable power simplifications.
`datafusion/functions/src/math/log.rs`	Guards nullable logarithm simplifications.
`datafusion/functions-aggregate/src/count.rs`	Marks `COUNT` order-insensitive.
`datafusion/core/tests/physical_optimizer/filter_pushdown.rs`	Expands filter-pushdown regression tests.
`datafusion/core/src/physical_planner.rs`	Forces null-aware hash joins.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…als in a filter A correlated `NOT IN` in anti join position was decorrelated into a null-aware anti join. On this branch the null-aware hash join takes a single key and checks the subquery's NULLs over every build row, so after equi-join extraction: - a constant `NOT IN` with an equality correlation got the correlation column as its only key, and the NULL rules applied to it: a NULL correlation key dropped rows although the subquery held no NULL; - a column `NOT IN` with an equality correlation had two keys and failed to plan ("null_aware anti join only supports single column join key"); - with a non-equality correlation, a subquery NULL that the correlation excludes still made every outer row UNKNOWN; - a constant `NOT IN` with a non-equality correlation had no key and failed to plan (the check apache#25348 added). In a filter, `x NOT IN (SELECT y ... WHERE c)` keeps a row exactly when no subquery row satisfying `c` has `x IS NULL OR y IS NULL OR x = y`. Build that filter and a plain anti join for a correlated `NOT IN`; an uncorrelated one keeps the null-aware hash join, which is correct for it. With non-nullable columns the disjunction simplifies to `x = y`, which stays an equi-join key. When the pull-up drops a subquery filter as a duplicate of the `IN` equality (`... WHERE y = x`), the subquery kept only values equal to `x`, none NULL, so the equality alone decides: `x = y` and the correlation, not null-aware. The null-aware plan got this wrong when no other correlation was left, treating the subquery as uncorrelated. `PullUpCorrelatedExpr` now records the drop. The `NOT EXISTS` form is not used when the pull-up grouped a scalar aggregate by the correlation, which loses the row the aggregate returns over no input, nor for a volatile `x`, which the filter would evaluate more than once. Those keep the null-aware plan, which does not support the aggregate shape. Upstream evaluates these shapes in the null-aware join executor instead (apache#25339, apache#25560), which builds on the null-aware mark joins of apache#21585 that this branch does not have. This is a Spice patch, to be dropped when the branch moves to a release that has them. `not_in_correlated.slt` checks 9 shapes over 9 datasets against SQLite's answers (46 of the 81 queries were wrong or failed to plan before this change, 0 after), the two dropped-duplicate shapes, and that the scalar aggregate shapes still fail to plan instead of answering from missing rows. The apache#25348 case that expected the planning error now expects the row SQL returns.

…c accumulator display The fork prints the `accumulator=` of a `HashJoinExec` (#117), so the plan this backport's EXPLAIN test pins shows it too.

## Which issue does this PR close? - Closes apache#23428. ## Rationale for this change Report correct nullability for `InSubquery` logical exprs. ## What changes are included in this PR? OR the expression's and the subquery's nullability, more like `ScalarSubquery`. ## Are these changes tested? 1. Targeted unit tests `InSubquery` nullability 2. Unit tests to verify the expression simplifier handles these queries correctly. ## Are there any user-facing changes? No --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com> (cherry picked from commit 70c26a0)

## Which issue does this PR close? - Closes apache#24513. ## Rationale for this change A scalar subquery that returns zero rows evaluates to `NULL`, even when its projected expression is non-nullable. DataFusion currently derives `Expr::ScalarSubquery` nullability from the subquery's output field. This can produce a non-nullable output schema containing `NULL`, and can cause `SimplifyExpressions` to incorrectly fold predicates such as: ```sql SELECT (SELECT 1 WHERE FALSE) IS NULL; ``` to `false`. This change deliberately marks all scalar subqueries nullable, including those guaranteed to return exactly one row (such as an ungrouped aggregate like `(SELECT count(*) FROM t)`). This is a conservative trade-off that gives up some nullability precision for correctness, and is consistent with how PostgreSQL treats scalar subqueries. A possible follow-up refinement is a `LogicalPlan::min_rows()` lower bound (mirroring the existing `max_rows()`), which would let uncorrelated scalar subqueries provably returning at least one row keep their projected field's nullability. ## What changes are included in this PR? Scalar subqueries are conservatively marked nullable in logical expression schema derivation and physical expression planning. The projected field's data type, name, and metadata are preserved. ## Are these changes tested? Yes. Unit tests cover logical schema derivation and expression simplification, and SQLLogicTests cover both zero-row execution and `IS NULL` correctness. The full workspace test suite and Clippy with warnings denied pass. ## Are there any user-facing changes? Yes. Zero-row scalar subqueries with non-nullable projections now return `NULL` without a schema validation error, and `IS NULL` predicates produce the correct result. In addition, output schemas containing scalar subqueries now always mark those fields as nullable, even for subqueries that can never produce `NULL`. Downstream consumers that inspect schema nullability can observe this change. No public APIs change. --- AI usage: Created with Claude Code and Opus 5. I have reviewed the code and made modifications where it made sense. --------- Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net> (cherry picked from commit 4798476)

Copilot

Copilot review overview

🔵 Needs a closer look

The large correctness backport includes correlated NOT IN behavior that conflicts with the documented scope and provenance.

Review effort: Balanced
Findings: 1

Open (1)

Document or remove the unsupported NOT IN decorrelation change · New

…ith a correlated NOT IN patch and scalar-subquery nullability Backported alone, apache/datafusion#25348 made correlated `NOT IN` worse on DataFusion 54.1 than on the current pin: a constant `NOT IN` with an equality correlation dropped rows on a NULL correlation key, and two more correlated shapes failed to plan. spiceai/datafusion#235 now plans a correlated `NOT IN` in a `WHERE` clause as the `NOT EXISTS` it equals there, and backports apache/datafusion#24516 (an empty scalar subquery is NULL) with its prerequisite #23429. Guards: `correlated_not_in_is_answered_as_not_exists`, `an_empty_scalar_subquery_is_null` and `an_in_subquery_over_a_nullable_column_is_nullable` in `fork_backport_guards.rs`, with their `docs/dev/fork_patches.md` rows.

…ports Repoint the [patch.crates-io] revs, Cargo.lock, fork_patches.md, and substrait-compliance pin at e9dc1dd4deed (spiceai/datafusion#235 merge on spiceai-54) and drop the TEMPORARY marker.

…ilter pushdown, simplification and planning (spiceai#14430) * fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning Pins spiceai/datafusion at spiceai/datafusion#235, which backports seventeen upstream fixes for queries that return wrong rows under Spice's defaults: dynamic filters built from the wrong aggregates or resolved onto a same-named column, `NOT IN` losing its NULL semantics to a pushed filter or a sort-merge join, null-equal joins and `INTERSECT` losing their NULL match, NULL-unsafe constant folding of `log`/`power`, `XOR` and `~ '.*'`, merged nested projections, `LIMIT … OFFSET` under a sort, `COUNT … ORDER BY`, the `TopK` aggregation dropping NULL groups, and cast statistics. Every fix gets a guard that runs its upstream regression query in a session built from Spice's default configuration and fails on the previous pin: `crates/cayenne/tests/datafusion_dynamic_filter_backports_test.rs` for the filter-pushdown fixes, over Parquet and over Cayenne tables, and `crates/runtime-datafusion/src/fork_backport_guards.rs` for the rest. `docs/dev/fork_patches.md` gains a row for each. The pin row stays marked TEMPORARY until the fork PR merges. * fix(substrait-compliance): report the DataFusion revision the harness is built against `DATAFUSION_FORK_REV`, which the harness prints and writes into each result's `datafusion_pin`, still named the previous pin, so results from this build would have been credited to it. The constant and the README's pin row now name the current revision, and a test fails whenever the constant and the workspace's `[patch.crates-io]` pin disagree, so the next move of the pin cannot leave it behind. * fix(deps): move the DataFusion pin to spiceai/datafusion#235's head with a correlated NOT IN patch and scalar-subquery nullability Backported alone, apache/datafusion#25348 made correlated `NOT IN` worse on DataFusion 54.1 than on the current pin: a constant `NOT IN` with an equality correlation dropped rows on a NULL correlation key, and two more correlated shapes failed to plan. spiceai/datafusion#235 now plans a correlated `NOT IN` in a `WHERE` clause as the `NOT EXISTS` it equals there, and backports apache/datafusion#24516 (an empty scalar subquery is NULL) with its prerequisite #23429. Guards: `correlated_not_in_is_answered_as_not_exists`, `an_empty_scalar_subquery_is_null` and `an_in_subquery_over_a_nullable_column_is_nullable` in `fork_backport_guards.rs`, with their `docs/dev/fork_patches.md` rows. * chore(deps): pin datafusion to spiceai-54 merge of wrong-results backports Repoint the [patch.crates-io] revs, Cargo.lock, fork_patches.md, and substrait-compliance pin at e9dc1dd4deed (spiceai/datafusion#235 merge on spiceai-54) and drop the TEMPORARY marker.

lyne7-sc and others added 23 commits September 26, 2026 09:13

Copilot AI balanced review requested due to automatic review settings September 26, 2026 22:30

lukekim self-assigned this Sep 26, 2026

Copilot started reviewing on behalf of lukekim September 26, 2026 22:31 View session

lukekim mentioned this pull request Sep 26, 2026

fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning spiceai/spiceai#14430

Merged

Copilot AI reviewed Sep 26, 2026

View reviewed changes

lukekim added 2 commits September 26, 2026 20:01

test: render the apache#22810 NOT IN plan with the fork's HashJoinExe…

1ef0eee

…c accumulator display The fork prints the `accumulator=` of a `HashJoinExec` (#117), so the plan this backport's EXPLAIN test pins shows it too.

AdamGS and others added 2 commits September 26, 2026 20:02

Copilot AI review requested due to automatic review settings September 27, 2026 03:06

Copilot started reviewing on behalf of lukekim September 27, 2026 03:07 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

Comment thread datafusion/optimizer/src/decorrelate_predicate_subquery.rs

krinart approved these changes Sep 27, 2026

View reviewed changes

lukekim merged commit e9dc1dd into spiceai-54 Sep 27, 2026
2 checks passed

lukekim deleted the lukim/spiceai-54-wrong-results-backports branch September 27, 2026 20:00

grokspice mentioned this pull request Sep 30, 2026

fix(deps): move the DataFusion pin past the four unparser fixes, and guard each of them (fixes #13022) spiceai/spiceai#14570

Merged

6 of 9 tasks

Navigation Menu

Repository navigation

fix: backport upstream fixes for wrong results from filter pushdown, dynamic filters, simplification and planning - #235

fix: backport upstream fixes for wrong results from filter pushdown, dynamic filters, simplification and planning#235
lukekim merged 27 commits into
spiceai-54from
lukim/spiceai-54-wrong-results-backports

lukekim commented Sep 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

Conversation

lukekim commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

lukekim commented Sep 26, 2026 •

edited

Loading