Repository navigation
[X-3985] Merge upstream 55.1.0 into branch-55 - #83
Closed
MassivePizza wants to merge 17 commits into
Closed
MassivePizza wants to merge 17 commits into
MassivePizza wants to merge 17 commits into
Conversation
…ize (apache#24384) (apache#24529) ## Which issue does this PR close? - Part of apache#24462 - Backport of apache#24384 to `branch-55` (for 55.1.0, tracked in apache#24462). - Fixes apache#24383 ## Rationale for this change `UnnestExec` emitted exactly one output batch per input batch, however many rows the unnesting produced, never consulting `datafusion.execution.batch_size`. This let downstream operators receive arbitrarily large batches and made peak memory scale with input batch size times list length instead of `batch_size`. No API change, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#24384 (commit 6187d47). No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage, all tests pass. ## Are there any user-facing changes? `UnnestExec` output batches now respect `datafusion.execution.batch_size` as an upper bound. No API changes. Co-authored-by: Andy Grove <agrove@apache.org>
…s adaptation (apache#24125) (apache#24530) ## Which issue does this PR close? - Part of apache#24462 - Backport of apache#24125 to `branch-55` (for 55.1.0, tracked in apache#24462). - Fixes apache#24109 ## Rationale for this change With `datafusion.execution.parquet.pushdown_filters = true`, a predicate on a struct field was reported as fully handled by the scan whenever the file needed schema adaptation, so `FilterExec` was removed from the plan and the predicate was silently dropped — returning every row instead of the filtered set. This is a correctness bug (wrong results), not specific to 55.0.0, so it fits the backport criteria. ## What changes are included in this PR? Cherry-pick of apache#24125 (commit 40c208e). Git's recursive merge auto-resolved surrounding context differences in `datafusion/physical-expr-adapter/src/schema_rewriter.rs` and `datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt`; no manual conflict resolution or adaptation of the fix itself was required. One follow-up commit adapts a test expectation: `branch-55` doesn't have apache#24130/apache#24315, which taught nested schema pruning to union the leaves needed by mixed whole-column + field-access reads (e.g. `select s, s['y'] from narrow`). Without that optimization, the mixed-access case falls back to reading every physical leaf, so `bytes_scanned` is `219` here instead of the `146` the original PR's test expects on `main`. This is a pre-existing difference in pruning capability, not a correctness regression from this fix. ## Are these changes tested? Yes. Carries the original regression coverage, all tests pass. ## Are there any user-facing changes? `WHERE s['field'] = ...` predicates on struct columns now filter correctly when Parquet filter pushdown requires schema adaptation. No API changes. --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
…te_physical_plan (apache#24690) (apache#24694) This PR is a cherry-pick of apache#24690 onto `branch-55`. See the original PR for a description of the issue. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uction (apache#24723) (apache#24752) This is a back port of apache#24723 onto `branch-55` to support `datafusion-python` upgrade to 55.1.0. The details can be found in the linked PR. This is needed for apache/datafusion-python#1677 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… to planned schema in aggregation (apache#24394) (apache#24699) This is a back port of apache#24394 for `branch-55`. The original issue is apache#24069 Note that the original PR was marked as an auto-detected api change, but I believe that is a false positive. --------- Co-authored-by: Patrick Ribbsaeter <patrickswedish@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… and new_default (apache#24582) (apache#24876) This is a back port of apache#24582 into `branch-55` for inclusion in 55.1.0. Please see the original PR for details. Co-authored-by: Unik Dahal <61407386+unikdahal@users.noreply.github.com>
This is a small update to the Cargo.lock file to update - chacha20 v0.10.0 -> v0.10.2 - h2 v0.4.13 -> v0.4.19 The first was due to a yanked crate. The second was due to a security vulnerability, RUSTSEC-2026-0258. The yanked crate is a warning in cargo audit and will auto-resolve on building to the newer 0.10.2, and the security vulnerability is already updated in `main`.
…asts (apache#23169) (apache#24875) This PR is a backport of apache#23169 onto `branch-55`. I made one update the `use` statement in `planner.rs`. Co-authored-by: Dewey Dunnington <dewey@wherobots.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…ache#24670) (apache#24992) Backports apache#24670 to branch-55. Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR updates the version number and changelog for the 55.1.0 release.
…che#25216) (apache#25485) - Fixes apache#25215 on branch-55 This PR: - Backports apache#25092 and apache#25216 from @comphead to the branch-55 line ## Rationale MinIO withdrew `minio/minio` from Docker Hub on 2026-09-11, so every `datafusion-cli` storage integration test on `branch-55` now panics while pulling the image (for example the CI failure on apache#25277). --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
…5309) (apache#25514) - Fixes the `security_audit` failure on `branch-55` (for example https://github.com/apache/datafusion/actions/runs/35435408307/job/105877099456 on apache#25277) This PR: - Backports apache#25309 from @Xuanwo to the branch-55 line ## Rationale `branch-55` locks `rustls` 0.23.39, which is affected by [RUSTSEC-2026-0285](https://rustsec.org/advisories/RUSTSEC-2026-0285.html) ("TLS 1.3 handshake messages incorrectly accepted across encryption level boundaries", published 2026-09-14). Every PR targeting `branch-55` now fails the `security_audit` job until the lockfile is updated. `rustls` 0.23.45 fixes the vulnerability. The cherry-pick applied cleanly: `branch-55` and `main` had identical `rustls`, `aws-lc-rs`, `aws-lc-sys`, and `rustls-webpki` entries before the fix. ## Testing `cargo audit` with the same ignores as the CI job passes on this branch (it fails on `branch-55` without this change): ```sh cargo audit --ignore RUSTSEC-2026-0194 --ignore RUSTSEC-2026-0195 ``` `cargo fetch --locked` and `cargo check --locked -p datafusion-cli` also succeed, confirming the updated lockfile resolves and builds. Co-authored-by: Xuanwo <github@xuanwo.io>
… (apache#25277) - Part of apache#24462 - Fixes apache#24474 on branch-55 This PR: - Backports apache#24351 from @dentiny to the branch-55 line ## Rationale As noted in apache#24462 (comment), publishing `datafusion-spark` for the 55.1.0 release required a manual local change because the crate does not compile without its optional `core` feature (see apache#24474). Backporting this fix means a future 55.2.0 release can be published without manual intervention. Verified locally that `cargo check -p datafusion-spark --no-default-features` now succeeds on this branch (it fails on `branch-55` without this change). Co-authored-by: dentiny <dentinyhao@gmail.com>
…e#25706) (apache#25759) ## Which issue does this PR close? - Fixes apache#25705 on `branch-55`. ## Rationale for this change This PR backports apache#25706 from @a10y to the `branch-55` line. `quay.io/minio/minio` no longer allows anonymous pulls. `branch-55` pulls that image for the `datafusion-cli` storage integration tests (it moved there from Docker Hub in apache#25485), so the `cargo test datafusion-cli (amd64)` job is expected to hit the same failure that apache#25705 reported on `main`. The last `branch-55` CI run was on 2026-09-19, before the image became unavailable, so no run on this branch has failed yet. ## What changes are included in this PR? A clean `git cherry-pick -x` of e2ca7f3 with no conflicts. The patch is identical to the original (same `git patch-id`). - `datafusion-cli/tests/cli_integration.rs`: run the S3 tests against a RustFS container (`docker.io/rustfs/rustfs:1.0.0`) instead of MinIO. - `.github/workflows/rust.yml`: pre-pull the RustFS image instead of the MinIO one. - `datafusion-cli/Cargo.toml`: drop the `minio` feature of `testcontainers-modules`. - `datafusion-cli/CONTRIBUTING.md`: update the storage test docs. ## What is the testing strategy for this PR? The existing `datafusion-cli` storage integration tests cover this change, and CI runs them in the `cargo test datafusion-cli (amd64)` job. On this branch, `datafusion-cli/tests/` and its snapshots match `main` apart from one unrelated Spark test. `testcontainers` and `testcontainers-modules` also resolve to the same versions as on `main`. Locally, `cargo fmt --all -- --check`, `cargo clippy -p datafusion-cli --all-targets --all-features -- -D warnings`, `taplo format --check` and the prettier docs check pass. The Docker-based storage tests were not run locally. CI runs them. ## Are there any user-facing changes? No. The change only affects tests, CI and contributor docs. Signed-off-by: Andrew Duffy <andrew@a10y.dev> Co-authored-by: Andrew Duffy <a10y@users.noreply.github.com>
…pache#24809) (apache#25821) ## Which issue does this PR close? - Fixes apache#24807 in the `branch-55` release line. - Part of the 55.2.0 release tracking issue apache#25758. - Backports the merged fix in apache#24809. ## Rationale for this change In DataFusion 55.1.0, running `EnsureRequirements` again on a plan with a pushed-down `fetch` can remove the global limit and return too many rows. This is a correctness fix for a bug present in the 55 release line, so it belongs in the 55.2.0 patch release. The incorrect result and expected behavior are described in apache#24807. ## What changes are included in this PR? - Cherry-picks the merged commit from apache#24809 onto `branch-55`. - Keeps fetched distribution operators as row-selection boundaries during reoptimization. - Preserves a fetched ordered merge's TopK selection when replacing order-preserving variants. - Carries over the original regression tests and adjusts the backport for the 55 branch. ## What is the testing strategy for this PR? - `cargo fmt --all` - `cargo test -p datafusion --test core_integration physical_optimizer::enforce_distribution` (87 passed) - `cargo clippy --all-targets --all-features -- -D warnings` - `./dev/rust_lint.sh` (passed with the CI-pinned `hawkeye 6.2.0`) - Initialized the repository's test-data submodules, then ran the documented extended workspace test command. One unrelated macOS RSS-threshold test failed: `smj_with_mem_limit_1_runner` measured 154.2 MB against a 152.6 MB allowance; it failed again in isolation at 158.6 MB. This test exercises sort-merge join memory accounting, which this backport does not change. - Ran the extended workspace suite with only that RSS test skipped: all `datafusion` library, integration, and fuzz tests passed. The custom SQLLogic runner does not accept libtest's `--skip` flag, so I ran all other workspace packages separately without a skip; those passed, including all 503 SQLLogic files. - No local benchmark directly exercises reoptimization of a fetched physical plan. ## Are there any user-facing changes? Queries keep their requested global limit when distribution requirements are optimized more than once. This corrects query results without changing public APIs.
Brings in the upstream 55.1.0 backports. Content matches rebasing the fork patches onto upstream/branch-55, with these fork commits superseded upstream: - #75 fix: preserve fetch across distribution reoptimization -> apache#24809 (backport apache#25821) - #79 fix(ci): pull the MinIO test image from quay.io -> apache#25092, apache#25216 (backport apache#25485), later replaced by the switch to rustfs in apache#25706 (backport apache#25759) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?