fix(deps): bump DataFusion for upstream fixes to wrong results from filter pushdown, simplification and planning

[lukekim]

📝 Summary

Bumps the spiceai/datafusion pin to e9dc1dd4de on spiceai-54, the merge of spiceai/datafusion#235, which backports eighteen upstream fixes for queries that return wrong rows and carries a Spice patch for correlated NOT IN. DataFusion 54.1 has none of them, and Spice's defaults reach each one: dynamic filter pushdown is on for joins, TopK and aggregates, and every Spice session sets parquet.pushdown_filters = true, so a Parquet scan applies a pushed filter row by row, as a Cayenne (Vortex) scan does for every predicate it accepts.

Filter pushdown and dynamic filters:

Upstream	Wrong result it fixes
apache/datafusion#24817	MIN(a), MAX(a), MAX(b), MIN(c + 1): the aggregate's dynamic filter came from the plain-column aggregates alone and pruned the rows holding MIN(c + 1)
apache/datafusion#25259	A join or TopK dynamic filter pushed below an operator with two same-named columns was resolved by name onto the wrong one: the join lost its only match, or ORDER BY p.amount LIMIT 1 returned the wrong row
apache/datafusion#23104	x NOT IN (subquery) returned rows although the subquery holds a NULL: the filter pushed into the subquery's scan dropped its NULLs
apache/datafusion#23173	x NOT IN (subquery) kept a NULL x when the pushed filter emptied the subquery's scan
apache/datafusion#22965, #23106	IS NOT DISTINCT FROM joins and INTERSECT lost their NULL = NULL match
apache/datafusion#22926, #24045	A mixed filter over an aggregate lost its aggregate-output predicate; a filter over an anti join was taken as handled by its non-output side. Both are prerequisites #25259 is written against, and reach Spice only when such a filter survives to the physical plan

Planning, simplification and execution:

Upstream	Wrong result it fixes
apache/datafusion#22810	With runtime.query.prefer_hash_join: false and more than one partition, NOT IN (subquery) was planned as a sort-merge join and a NULL in the subquery stopped excluding rows
apache/datafusion#23684	The TopK aggregation dropped groups whose MIN/MAX is NULL
apache/datafusion#24247, #24248, #24380	log(a, 1), power(a, 0), a XOR a, col ~ '.*' and their kin were folded to constants, wrong where the input is NULL
apache/datafusion#24686	Merging nested projections that redefine a column used the wrong layer: six i + 1 layers added three
apache/datafusion#24958	ORDER BY over a LIMIT … OFFSET subquery returned too few rows
apache/datafusion#24997	COUNT(a, c ORDER BY b) counted only a, and a grouped COUNT(a ORDER BY b) panicked
apache/datafusion#25348	3 NOT IN (subquery) returned rows although the subquery holds a NULL
apache/datafusion#25227	MIN/MAX over a cast column was answered from uncast Parquet statistics
apache/datafusion#24516	(SELECT 1 WHERE FALSE) IS NULL answered false: a scalar subquery that returns no rows is NULL, but was typed with its column's nullability

The fork branch also carries #24428 (a FilterRemapper fast path #25259 builds on), #23429 (the IN subquery nullability #24516 builds on) and five test-only adaptations to 54.1. Every backport is a git cherry-pick -x, with each conflict resolution recorded in its commit message; spiceai/datafusion#235 lists them.

A Spice patch for correlated NOT IN. Backported alone, #25348 made correlated NOT IN worse on 54.1 than it is on the current pin. Its null-aware anti join takes one key and checks the subquery's NULLs over every build row. So a NULL correlation key dropped rows although the subquery held no NULL, a shape the current pin answers correctly. A subquery NULL that the correlation excludes still removed every outer row. Two more shapes failed to plan: a column NOT IN with an equality correlation, and a constant one with a non-equality correlation. The second is the planning error this PR used to introduce.

Upstream fixes these in the null-aware join executor (#25339, #25560), on top of null-aware mark joins 54.1 does not have. spiceai/datafusion#235 instead plans a correlated NOT IN in a WHERE clause as the NOT EXISTS it equals there: a plain anti join on x IS NULL OR y IS NULL OR x = y and the correlation. An uncorrelated NOT IN keeps the null-aware hash join. Measured against SQLite on 81 queries (9 NOT IN shapes over 9 datasets with NULLs in the outer value, the subquery's values and the correlation keys):

	wrong rows	planning error
current pin (11624fb82d)	20	18
#235 with #25348 alone (5ac6b8edb5)	19	27
#235 (d1815c2a37, merged as e9dc1dd4de with the same tree)	0	0

A scalar aggregate that the decorrelation groups by the correlation keeps the plan it has on the current pin, which fails to plan, because the grouped form loses the row the aggregate returns over no input. A volatile x also keeps its current plan.

Not backported, and still wrong on the fork: #24932 and #25234, which build on larger upstream changes (EnsureRequirements, a time ± interval result-type change), and a NOT IN under OR or IS NULL, which is evaluated by a mark join that is not null-aware on 54.1 (#25560, on #21585).

Reproduction

Every guard runs its fix's upstream regression query in a session built from Spice's default DataFusion configuration (runtime_datafusion::session_config::get_df_default_config). crates/cayenne/tests/datafusion_dynamic_filter_backports_test.rs covers the filter-pushdown fixes, over Parquet files and, where the shape is reachable, over Cayenne tables whose rows are in Vortex files. crates/runtime-datafusion/src/fork_backport_guards.rs covers the rest. On the current pin (11624fb82d), all 21 fail:

"Parquet: SELECT MIN(a), MAX(a), MAX(b), MIN(c + 1) FROM agg_dyn_mixed: got [\"1,8,12,101\"], expected [\"1,8,12,71\"]",
"Cayenne: SELECT MIN(a), MAX(a), MAX(b), MIN(c + 1) FROM agg_dyn_mixed: got [\"1,8,12,91\"], expected [\"1,8,12,71\"]",
"SELECT s.id AS sid, a.id AS aid, b.id AS bid FROM issue_25244_b s JOIN (SELECT a.id, b.id FROM issue_25244_a a LEFT JOIN issue_25244_b b ON a.ty = b.id GROUP BY a.id, b.id) ON s.id = b.id: got [], expected [\"x1,a1,x1\"]",
"SELECT o.amount, p.amount FROM issue_25296_orders o JOIN issue_25296_payments p ON o.id = p.id JOIN issue_25296_customers c ON p.id = c.id ORDER BY p.amount LIMIT 1: got [\"100,30\"], expected [\"300,10\"]",
"Cayenne: SELECT id FROM asa_outer WHERE id NOT IN (SELECT eid FROM asa_inner): got [\"1\", \"3\"], expected []",
"Cayenne: SELECT id FROM ao WHERE id NOT IN (SELECT eid FROM i_disj): got [\"5\", \"NULL\"], expected [\"5\"]",
"Parquet: SELECT id FROM nej_build INTERSECT SELECT id FROM nej_probe: got [\"11\"], expected [\"11\", \"NULL\"]",
"SELECT count(*) FROM (SELECT a, b, count(b) AS cnt FROM agg_filter_pushdown GROUP BY a, b) q WHERE cnt = 2 AND b = 'foo': got [\"1\"], expected [\"0\"]",
"SELECT count(*) FROM join_left l LEFT ANTI JOIN right_parquet r USING (id) WHERE false: got [\"5\"], expected [\"0\"]",
test result: FAILED. 0 passed; 8 failed          (cayenne datafusion_dynamic_filter_backports_test)

not_in_stays_a_hash_join_when_sort_merge_joins_are_preferred   left: Ok(["1", "3", "4"])                    right: Ok([])
topk_aggregation_keeps_groups_whose_max_is_null                left: Ok(["30", "20", "10"])                 right: Ok(["30", "20", "10", "NULL"])
log_and_power_simplification_keeps_null                        left: Ok(["false,false,false,false,false"])  right: Ok(["true,true,true,true,true"])
xor_simplification_keeps_null                                  left: Ok(["7,7,0", "7,7,0"])                 right: Ok(["7,7,0", "NULL,NULL,NULL"])
regex_match_all_simplification_keeps_null                      left: Ok(["foo,true", ",true", "NULL,false"]) right: Ok(["foo,true", ",true", "NULL,NULL"])
nested_projections_keep_every_layer                            left: Ok(["6", "7", "8"])                    right: Ok(["10", "11", "9"])
a_sort_over_limit_offset_keeps_every_row                       left: Ok(1)                                  right: Ok(4)
count_with_order_by_counts_every_argument                      left: Ok(["2"])                              right: Ok(["3"])
constant_not_in_sees_the_subquerys_null                        left: Ok(["1", "2"])                         right: Ok([])
a_cast_aggregate_is_not_answered_from_uncast_statistics        left: Ok(["1,2"])                            right: Ok(["1,100"])
correlated_not_in_is_answered_as_not_exists                    3 NOT IN (… t2.g = t1.g):   left: Ok(["1", "2", "3", "NULL"])  right: Ok(["1", "3", "NULL"])
                                                               x NOT IN (… t2.g = t1.g):   left: Err("Error during planning: null_aware anti join only supports single column join key, got 2 columns")  right: Ok(["3"])
                                                               3 NOT IN (… t2.g > t1.g):   left: Ok(["1", "2", "3", "NULL"])  right: Ok(["2", "3"])
                                                               x NOT IN (… t2.g > t1.g):   left: Ok([])                       right: Ok(["3"])
an_empty_scalar_subquery_is_null                               left: (Ok(["false"]), Ok([]))            right: (Ok(["true"]), Ok(["1", "2"]))
an_in_subquery_over_a_nullable_column_is_nullable              "an IN over a subquery that can hold a NULL is nullable" (the IN column is typed non-nullable)
test result: FAILED. 0 passed; 13 failed         (runtime-datafusion fork_backport_guards)

On this branch: test result: ok. 8 passed and test result: ok. 13 passed.

The three guards for the correlated NOT IN patch and #24516 also fail on #235 without them (5ac6b8edb5), where the correlated constant NOT IN with an equality correlation answers Ok([]) and its non-equality form fails to plan with "null_aware LeftAnti join requires equi-join keys, but the join has none".

The full gate on the pinned head dec5d25c37 (remote sign-off): 14189 tests run: 14189 passed (2 slow, 1 flaky), 134 skipped, all 21 guards included, and scripts/check_fork_patches.py reports fork-patch ledger: 31 pinned forks, all recorded. The flaky test, cayenne's schema_evolution_live_decimal_scale_change_drops_statistics, failed its first try with the same panic on trunk's pin (11624fb82d) in two other branches' gates the same day.

📚 Docs

docs/dev/fork_patches.md: a row and a guard for each of the eighteen fixes, #23429 and the correlated NOT IN patch, and a note that fixes cherry-picked from upstream main are carried like Spice patches until a re-cut lands on a release that has them. The pin row names e9dc1dd4de on spiceai-54.

tools/substrait-compliance: the harness's DATAFUSION_FORK_REV and the README's pin row name the new pin, and a test now fails when they disagree with the workspace pin, so the harness cannot credit its results to a revision it was not built against.

👀 Notes for Reviewers

• The pin is e9dc1dd4de, the rebase-merge of fix: backport upstream fixes for wrong results from filter pushdown, dynamic filters, simplification and planning datafusion#235 on spiceai-54. Its tree is identical to the tested head d1815c2a37's (d6cfcb2cfd), so the measurements above hold for it.
• spiceai-54 has since merged four unparser fixes (fix(unparser): scope a filter on a projection output that cannot be repeated, gated by dialect (refs spiceai/spiceai#12751) datafusion#227, Update to use correct data-components-contrib commit #230, Can't get 'spice run' #231 and v0.1.1-alpha Endgame #232, for An unnamed projection output stays unbindable when the projection becomes a derived table #12751, Unparser emits wrong SQL for a RightMark join: wrong relation, missing mark column, no EXISTS #13022, Federated FULL OUTER JOIN still drops rows when the filtered input is a join rather than a bare scan #12593 and unparser: an EXISTS build-side join key can lose its binding and escape to the outer query #13493). This pin leaves them out; they move the pin in a PR of their own, with their ledger rows and guards.
• Beyond the rows that change, a plan for a mixed MIN/MAX aggregate no longer carries a dynamic filter, and a null-equal join's pushed filter gains OR key IS NULL.

[github-actions]

✅ Pull with Spice Passed

Passing checks:

• ✅ Title meets minimum length requirement (10 characters)
• ✅ No banned labels detected
• ✅ Has a label from required category kind/
• ✅ Has a label from required category area/
• ✅ Has at least one assignee: lukekim

[Copilot]

Copilot review overview

🟡 Changes recommended

The Substrait compliance harness still attributes results to the previous DataFusion revision.

Review effort: Balanced
Findings: 1

Open (1)

• Update compliance harness revision pin and documentation · New

What changed in this PR

Updates DataFusion to a fork revision containing correctness backports for wrong SQL results.

Changes:

• Bumps all DataFusion dependencies to the backport revision.
• Adds 18 regression guards covering Parquet and Cayenne execution.
• Documents each fork patch and its guard.

File	Description
Cargo.toml	Updates DataFusion workspace pins.
Cargo.lock	Locks the new revisions and test dependency.
docs/​dev/​fork_patches.md	Records backports and regression guards.
crates/​runtime-datafusion/​Cargo.toml	Adds tempfile for tests.
crates/​runtime-datafusion/​src/​lib.rs	Registers the guard module.
crates/​runtime-datafusion/​src/​fork_backport_guards.rs	Tests planning, simplification, and execution fixes.
crates/​cayenne/​tests/​datafusion_dynamic_filter_backports_test.rs	Tests pushdown fixes against Parquet and Cayenne.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[lukekim]

Regression run on this branch (pin 5ac6b8edb5), cargo test --profile bench -p cayenne --lib --tests --no-fail-fast and -p runtime-datafusion --lib:

• runtime-datafusion library: 305 passed.
• cayenne integration tests: all 86 test binaries pass.
• cayenne library: 1420 passed, 2 failed under the full parallel run. Rerun alone:
  • metastore::sqlite::tests::test_writers_are_granted_the_write_lock_in_arrival_order: 3 of 3 pass.
  • provider::table::tests::pk_point_lookup_warm_concurrent_throughput_reuses_listing_and_scan_view asserts a wall-clock speedup (eight workers at least 2× faster than one serialized). It passed 8 of 9 runs on this pin and 8 of 9 on the previous pin (11624fb82d), so its occasional failure predates this change.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The correctness-critical optimizer backports require final human review after the upstream fork PR merges and the temporary pin is replaced.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Update compliance harness revision pin and documentation

[Copilot]

Copilot review overview

🔵 Needs a closer look

Correctness-critical optimizer backports require human validation after the SQL-surface and regression-guard issues are addressed.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Pin rejects valid correlated constant NOT IN queries · New
• Guard misses non-equality correlated NOT IN safety coverage · New

[Copilot]

Copilot review overview

🟡 Changes recommended

The pin targets an unmerged branch and currently fails the repository’s fork-patch landing guard.

Review effort: Balanced
Findings: 1

Open (1)

• DataFusion remains pinned to an unlanded fork branch · New

Resolved since last review (2)

• Pin rejects valid correlated constant NOT IN queries
• Guard misses non-equality correlated NOT IN safety coverage

[lukekim]

Regression run on the new pin (d1815c2a37, b2885cd):

• cargo test --profile bench -p runtime-datafusion --lib: test result: ok. 308 passed; 0 failed. That is the 305 from the previous head plus the three new guards.
• cargo test --profile bench -p cayenne --lib --tests --no-fail-fast: 86 of 87 binaries pass. The library binary had 1420 passed; 2 failed, from the same two load-sensitive tests that fail under this run's parallel load on the other branches too: metastore::sqlite::tests::test_writers_are_granted_the_write_lock_in_arrival_order and provider::table::tests::pk_point_lookup_warm_concurrent_throughput_reuses_listing_and_scan_view. Rerun on their own on this pin: test result: ok. 2 passed; 0 failed.

On the fork, the full sqllogictest suite fails the same 27 files at the same queries as on spiceai-54 (11624fb82), so this branch adds no failure there.

[Copilot]

Copilot review overview

🟡 Changes recommended

The intentionally temporary branch-head pin fails the repository landing guard until the upstream fork PR merges.

Review effort: Balanced
Findings: 1

Open (1)

• DataFusion remains pinned to an unlanded fork branch

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.