Repository navigation
55.1 upgrade - #181
Draft
gene-bordegaray wants to merge 891 commits into
Draft
55.1 upgrade#181gene-bordegaray wants to merge 891 commits into
gene-bordegaray wants to merge 891 commits into
Conversation
## Rationale for this change Logical `CASE` nullability unwraps null preserving casts before analyzing guarded branches, but physical `CASE` nullability did not. Type coercion could therefore produce conflicting schemas and cause valid aggregation queries to fail during planning --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…driven optimizations (apache#23821) ## Which issue does this PR close? This PR adds test coverage rather than closing an issue. It documents the current behavior of these bugs so that fixing them shows up as a test change: - apache#23634 - apache#23818 - apache#23819 - apache#23820 ## Rationale for this change While reviewing apache#23636 @neilconway and I kept coming up with more examples of bad plans, and it was not obvious which of the surrounding wrong answers were pre-existing and which the PR introduced. ## What changes are included in this PR? A new `datafusion/sqllogictest/test_files/functional_dependencies.slt` with one section per consumer of functional dependencies: 1. `ReplaceDistinctWithAggregate` — removing `DISTINCT` 2. `eliminate_duplicated_expr` — dropping trailing `ORDER BY` keys 3. `optimize_projections` — dropping `GROUP BY` expressions 4. `add_group_by_exprs_from_dependencies` — selecting non-grouped columns 5. `GROUP BY` derived keys on the NULL-padded side of an outer join Cases that currently return wrong answers are labelled `BUG` with the expected result and a link to the issue: | Case | Symptom | Issue | | --- | --- | --- | | 1.2 | `DISTINCT` over a nullable `UNIQUE` column returns both `NULL` rows | apache#23634 | | 2.2 | `ORDER BY x, y` drops the `y` key, so the `NULL` rows come back unordered | apache#23818 | | 3.2 | `GROUP BY x, y` drops `y`, merging the two `NULL` groups and losing a row | apache#23819 | | 4.2 | `SELECT x, y ... GROUP BY x` returns two rows for the `x = NULL` group | apache#23820 | ## Are these changes tested? CI ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…3679) ## Which issue does this PR close? - Closes: N/A ## Rationale for this change There are a reasonable number of places where we have functions marked as `async` that don't need to be so. Fix this by enabling the `unused_async` lint, and fixing up the resulting breakage. There are a handful of spots that need an `expect(clippy::unused_async)` -- mostly mocks of `async` methods and example code. ## What changes are included in this PR? * Enable `unused_async` lint * Remove unnecessary `async` annotations in a bunch of places * Add `expect(clippy::unused_async)` where necessary ## Are these changes tested? Yes, covered by existing tests (no behavioral change expected). ## Are there any user-facing changes? Yes: this PR updates a few public APIs: - `datafusion_cli::command::OutputFormat::execute` - `datafusion::test_util::parquet::TestParquetFile::create_scan` - `datafusion_datasource_csv::file_format::CsvFormat::read_to_delimited_chunks_from_stream` - `datafusion_substrait::serializer::deserialize_bytes` Migration is mostly straightforward (e.g., removing `await` from calling code).
…fusion/wasmtest/datafusion-wasm-app (apache#23868) Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 5.2.6 to 6.0.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/releases">webpack-dev-server's releases</a>.</em></p> <blockquote> <h2>v6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers import the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build, allowing the package to be consumed from both ESM and CommonJS without relying on <code>require(ESM)</code> for CommonJS consumers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md">webpack-dev-server's changelog</a>.</em></p> <blockquote> <h2>6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers <code>import</code> the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build — so the package works from both ESM and CommonJS, including environments where <code>require(ESM)</code> is not supported. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/webpack/webpack-dev-server/commit/05cb7921b2cd216f8caa74d2bdcc39b3f0d05bea"><code>05cb792</code></a> chore(release): new release (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5692">#5692</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/a451839256733b37c8b71d15b9e22a55db3290d1"><code>a451839</code></a> fix: handle middleware teardown in plugin mode (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5703">#5703</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/c2d23a7e9f8562c634b3f3cabf8cee18123a402b"><code>c2d23a7</code></a> fix: load ESM-only dependencies with native import() in the CommonJS build (#...</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/ba54764c65464b43ab695ab8e7f20ca8e6c92e8d"><code>ba54764</code></a> fix: reject cross-site requests to open-editor and invalidate endpoints (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5691">#5691</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2b369b39d537331fc44df3d3c1d62f2ac4598016"><code>2b369b3</code></a> fixup!</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/08a0ea7e92b2f954d2ab88e4b9669a45108cf0c8"><code>08a0ea7</code></a> fix: ensure undefined options default to an empty object in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/797b9e7d780eb5202895021bbd47253774a0135d"><code>797b9e7</code></a> fix: handle undefined options in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/e90221cd422dbaa71b3a870538be1c88caf98361"><code>e90221c</code></a> feat: plugin support (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5650">#5650</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/4c351e1b45f6a0bcc5fb3ef1e6bfad2d0c4fdd8b"><code>4c351e1</code></a> feat: support universal platform as a web target (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5690">#5690</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2236aa4b490a0b6abfd9e234beb3da5a32da91e7"><code>2236aa4</code></a> chore: update http-proxy-middleware to version 4.1.1 and add tests for pathRe...</li> <li>Additional commits viewable in <a href="https://github.com/webpack/webpack-dev-server/compare/v5.2.6...v6.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#23507. ## Rationale for this change Part of epic apache#23494. Moves `HashJoinExec` protobuf serialization from central dispatch. ## What changes are included in this PR? Add protobuf serialization and deserialization to the `HashJoinExec` physical plan implementation and deprecate the corresponding central proto methods. The wire format remains unchanged. ## Are these changes tested? Yes, existing HashJoin round-trip tests cover this plan. ## Are there any user-facing changes? Existing methods are deprecated with no immediate API change.
…che#23825) Closes apache#23514. Part of apache#23494. Migrate `AsyncFuncExec` proto encode/decode into the plan itself via `try_to_proto` / `try_from_proto`, removing its central-arm handling in `physical_plan/mod.rs`. ## Rationale `datafusion-proto` currently downcasts `AsyncFuncExec` in the central encode match and rebuilds it inline on decode. apache#23495 introduced self-serializing hooks so each plan owns its own wire format. This moves `AsyncFuncExec` onto that pattern. ## Changes - Added `try_to_proto` / `AsyncFuncExec::try_from_proto`, wired into the decode dispatch - Removed the central encode downcast branch - Old helper methods kept as `#[deprecated]` stubs, per existing convention - Wire format unchanged ## Testing Existing `roundtrip_async_func_exec` integration test now exercises the new hooks (old path deleted, so it's the only path left). `cargo fmt` + `cargo clippy --all-targets --features proto -- -D warnings` clean. ## User-facing changes No Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23513. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Window plans still relied on centralized protobuf dispatch, keeping serialization separate from the execution plans that own their state. Both window executors share one protobuf variant. Decoding must inspect `input_order_mode` before selecting the concrete plan. Add plan-local encoders for both executors and a decoder for the shared node. Keep the legacy helpers as deprecated delegates while preserving window frames and UDF payloads on the existing wire format. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23511. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Physical plan protobuf serialization is being moved from the central dispatch module into each `ExecutionPlan` implementation. Co-locating this logic with the corresponding execution plan makes the serialization code easier to maintain and incrementally reduces the central downcast chain. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement plan-local protobuf serialization and deserialization for `ExplainExec`. - Implement plan-local protobuf serialization and deserialization for `AnalyzeExec`. - Remove both plans from the live central serialization dispatch. - Retain the deprecated compatibility methods as delegates to the new implementations. - Preserve the existing protobuf wire format. - Strengthen roundtrip tests to cover all stored fields, including every `StringifiedPlan` variant, metric categories, and explain formats. Each execution plan migration is kept in a separate commit. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. The following checks pass: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - Full extended workspace test suite - `cargo test -p datafusion-proto --test proto_integration` The proto integration suite passes all 209 tests after rebasing onto the latest `main`. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No. This is an internal refactor and does not change the protobuf wire format or user-facing behavior. --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23512. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Aggregate protobuf conversion lived in the central proto crate, which prevented AggregateExec from owning its function-codec serialization. Move encoding and decoding into AggregateExec and keep the deprecated helpers as delegates so existing callers retain wire-compatible behavior. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
…nparser (apache#23789) ## Which issue does this PR close? - Closes apache#23668. ## Rationale for this change - When printing a GROUP BY query, the unparser sometimes wraps the aggregate's input in an inner subquery (`... FROM (SELECT ...)`). - Table aliases like `cs` only exist inside that subquery. Any clause outside it (SELECT, GROUP BY, HAVING, QUALIFY, ORDER BY) must use the subquery's output columns, not the aliases, so the unparser drops the alias. - Each clause does that dropping on its own, so it is easy to miss one. - ORDER BY on an aggregate that is not in the SELECT list was missed: it printed `ORDER BY round(sum("cs"."total_revenue"), 2)`, while the SELECT list in the same query correctly printed `sum("total_revenue")`. - DataFusion reads that SQL back fine, but stricter databases reject it because `cs` is out of scope there. ## What changes are included in this PR? I moved the rule into one helper, `UnparserAggScope`, that checks once per aggregate whether the input is an inner subquery and then prepares expressions for each clause. SELECT, GROUP BY, HAVING, QUALIFY, and both ORDER BY paths now go through it, which fixes the ORDER BY case. The window-over-aggregate path is left as-is with a comment: it is only reachable from hand-built plans, since a window in SQL always sits inside a SELECT that already drops the alias. Only ORDER BY output changes; every already-correct case stays the same. ## Are these changes tested? Yes. New tests in `datafusion/core/tests/sql/unparser.rs` cover the inner-subquery shape for a window sorting by an aggregate, ORDER BY on an unselected aggregate (the fixed case), and a top-level ORDER BY (the second sort path). Existing unparser tests and the TPC-H/Clickbench roundtrips still pass. ## Are there any user-facing changes? ORDER BY over an aggregate is now printed without an out-of-scope table qualifier, so the generated SQL is valid for stricter databases. No API changes.
…che#22980) ## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
… spark (apache#23766) ## Which issue does this PR close? N/A ## Rationale for this change Hex encoding was implemented six times across the workspace, at two different levels of optimization. Three copies used a fast byte-pair lookup table: - `datafusion/spark/src/function/math/hex.rs` - `datafusion/functions/src/string/to_hex.rs` - `datafusion/functions/src/encoding/inner.rs` (via the `hex` crate) The other three used a slower nibble-at-a-time loop pushing one character at a time: - `datafusion/functions/src/crypto/md5.rs` - `datafusion/spark/src/function/hash/sha1.rs` - `datafusion/spark/src/function/hash/sha2.rs` Beyond the duplication, this split meant the digest functions were paying for a slower encoder than the one already sitting elsewhere in the tree. Consolidating on a single implementation removes the duplication and moves `md5`, `sha1`, and `sha2` onto the fast path. ## What changes are included in this PR? A new `datafusion_common::utils::hex` module holds the only hex encoder in the workspace: ```rust pub enum HexCase { Lower, Upper } pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>); pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String; pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]); pub fn encode_u64(v: u64, case: HexCase, buf: &mut [u8; 16]) -> &[u8]; ``` Four entry points rather than one because the call sites have genuinely different output needs, and forcing them through a single shape would cost an allocation somewhere: `to_hex` writes straight into a `StringArray` values buffer, Spark's `hex` appends to a reused scratch `Vec`, `encode` writes into a pre-sized slice, and the digest functions want an owned `String`. Migrated call sites, all producing bit-identical output: | File | Change | | --- | --- | | `functions/src/string/to_hex.rs` | local table and two write helpers deleted; eight trait impls collapsed to two macros | | `spark/src/function/math/hex.rs` | two nibble tables, two lookup tables, `build_hex_lookup` and `hex_int64` deleted | | `functions/src/crypto/md5.rs` | local table and `hex_encode` deleted | | `spark/src/function/hash/sha1.rs` | local table and inline nibble loop deleted | | `spark/src/function/hash/sha2.rs` | local table and `hex_encode` deleted; eight call sites migrated | | `functions/src/encoding/inner.rs` | both encode sites moved off the `hex` crate | Deliberately left alone: - `hex::decode` / `hex::decode_to_slice` in `encoding/inner.rs`, and Spark's `unhex`. The decode direction has different semantics — Spark's `unhex` left-pads odd-length input, the `hex` crate does not — so unifying it is a separate question. The `hex` dependency stays for those. - `ScalarValue`'s binary `Display` impl in `common/src/scalar/mod.rs`, which writes to a `fmt::Formatter` rather than a byte buffer, and is a cold path. Three details worth a reviewer's attention: - The two ancestors of `encode_u64` disagreed on zero: `to_hex` wrote `'0'` into the caller's buffer, Spark's `hex_int64` returned a `'static` `b"0"` that never touched it. The shared version always writes into the buffer and returns a subslice of it, so the lifetime is uniform. Both callers still produce `"0"`. - Spark's `hex_encode_bytes` guards large binary input with `checked_mul(2)` + `try_reserve`, returning a `DataFusionError` rather than aborting on allocation failure. That guard stays at the call site; `encode_bytes_into` performs no reservation of its own, so behaviour is unchanged. - The encoders are `#[inline(always)]`, not `#[inline]`. They are called once per row from two other crates, and plain `#[inline]` left them out-of-line across the crate boundary. Measured cost of that: +3% on Spark's byte paths and up to +18% on `to_hex`'s i32 path, i.e. the refactor was a net regression on those benchmarks until the attribute changed. ## Are these changes tested? Every migrated function keeps its existing unit and sqllogictest coverage, which is what pins bit-identical output — in particular `spark/hash/sha1.slt` and `sha2.slt` assert concrete digest strings for all four SHA-2 bit lengths across both the scalar and array paths, and `expr.slt` covers `to_hex` and `md5`. New unit tests in `common/src/utils/hex.rs` cover zero, `u64::MAX`, single-nibble values, the odd/even digit-count boundary, two's complement of negative input, empty input, all 256 byte values in both cases, appending into a non-empty buffer, and that a reused scratch buffer never leaks stale digits between calls. Tests cross-check against `format!("{:x}")` rather than restating the implementation. Two Spark tests changed. `test_hex_int64` now drives `hex_encode_int64` instead of the deleted private `hex_int64`, keeping all ten cases including `i64::MIN`, `-1`, and the uppercase expectations. `test_hex_lookup_table_covers_all_bytes` was deleted — it only cross-checked the raw lookup tables, and `encode_bytes_covers_every_byte_value` now does that exhaustively through the public API. A test was added for Spark's lowercase byte path, which previously had no coverage. ### Benchmarks Criterion, `apache/main` @ `eef101769` as baseline. Median of the reported change interval. `datafusion/functions/benches/to_hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `i32_random` | −13.8% | −12.8% | −10.2% | | `i64_random` | −8.8% | −9.5% | −8.1% | | `i64_large_values` | −9.9% | −9.8% | −7.9% | `scalar_i32` −2.3%, `scalar_i64` −1.8%. `datafusion/spark/benches/sha2.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `array_binary_256` | −20.7% | −18.2% | −19.1% | | `array_scalar_binary_256` | −14.6% | −13.2% | −13.1% | `scalar/size=1` −3.3%. `datafusion/spark/benches/hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `hex_int64` | −2.9% | −3.6% | −3.5% | | `hex_int64_dict` | −3.2% | −1.5% | −0.2% | | `hex_utf8` | −0.2% | −0.3% | +0.7% | | `hex_binary` | −0.4% | −0.6% | −0.3% | The `hex_utf8` and `hex_binary` paths already used the byte-pair table before this PR, so they are expected to be flat; they are. `datafusion/functions/benches/crypto.rs`: `md5_array` −4.0%, `md5_scalar` −3.7%. The `sha224` / `sha256` / `sha384` / `sha512` cases in that file range from −0.1% to +2.2%, but they exercise `crypto/basic.rs`, which this PR does not modify and which contains no hex encoding — those numbers are run-to-run variance, not an effect of this change. No number is quoted for Spark `sha1`: it has no benchmark, and its change is the same substitution applied to `md5` and `sha2`. ## Are there any user-facing changes? No behaviour change — all migrated functions produce byte-identical output. `datafusion_common::utils::hex` is new public API on `datafusion-common`. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…t_batch` memory (apache#23873) ## Which issue does this PR close? - related to apache#23716 ## Rationale for this change It adds test coverage for two gaps found while reviewing apache#23716. ## What changes are included in this PR? Tests only, no functional change. - Dictionary inputs - memory usage on retract (make sure memory is released) Note I moved `array_agg` cases out `aggregate.slt` as it is already more than 9k lines long ## Are these changes tested? They are only tests ## Are there any user-facing changes? No. Tests only, no public API changes. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…3881) ## Which issue does this PR close? N/A ## Rationale for this change Two Spark functions allocated a `String` per row purely to render a small, bounded amount of text: - `bin` called `format!("{value:b}")` for every row. - `char` called `ch.to_string()` for every row — a heap allocation for a single character. In both cases the output has a known upper bound (64 binary digits for an `i64`, 4 bytes for a UTF-8 character), so the rendering fits in a stack buffer and the result can be appended straight to a pre-sized builder. ## What changes are included in this PR? `math/bin.rs`: - `spark_bin` now writes digits right-aligned into a caller-supplied `[u8; 64]` and returns a `&str` borrowed from it, instead of returning an owned `String`. - The `collect::<StringArray>()` becomes an explicit loop over a `StringBuilder::with_capacity`, sized at 8 digits per row. - Negative values still render as their two's-complement bit pattern, matching `{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as `"0"` rather than the empty string. `string/char.rs`: - `ch.to_string()` becomes `ch.encode_utf8(&mut encoded)` against a `[u8; 4]` hoisted out of the loop. Output is unchanged in both cases. ## Are these changes tested? Existing coverage pins the behaviour. `spark/math/bin.slt` asserts concrete output for the cases the rewrite had to get right: zero, negative values, `i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` / `-32768` widened from narrower integer types. `spark/string/char.slt` covers the negative-input empty string, the null path, and characters on both sides of the ASCII boundary (`char(256)` and above wrap via `% 256`). All 119 `spark/math` and `spark/string` sqllogictest files pass, along with the 258 `datafusion-spark` unit tests. The `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is added separately in apache#23882 so the baseline can be measured on `main` before this change lands. It covers 1024 and 8192 rows with 20% nulls over two value distributions: small values that render to a handful of digits, and full-range values that render to the maximum 64. `char` already had `datafusion/spark/benches/char.rs` on `main`, so no benchmark change is needed for it. ### Benchmarks Criterion, `apache/main` @ `f1ab86dad` as baseline. Median of the reported change interval. | Benchmark | 1024 | 8192 | | --- | --- | --- | | `bin/small` | −75.7% | −73.7% | | `bin/wide` | −47.8% | −49.8% | `char` (1024 rows): −76.1%. ## Are there any user-facing changes? No. Both functions produce byte-identical output; this is purely an allocation change.
…ts enabled (apache#23848) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#23847 . ## Rationale for this change Fixes a correctness bug, not sure when that config used/desirable. This is another thing I ran into during apache#21585 ## What changes are included in this PR? 1. New SLT test 2. Fix for bug in `datafusion-optimizer` ## Are these changes tested? Existing tests, and a new SLT test verifying the config change doesn't affect the query result and basic unit test. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
…lter (apache#23901) ## Which issue does this PR close? - Closes apache#23900. ## Rationale for this change `push_down_filter` infers equi-key predicates across a join's ON keys and pushes them to the opposite side. For a null-aware join (the `LeftAnti` join produced by `NOT IN` with a nullable subquery), an outer predicate on the left key like `outer.id > 5` is rewritten to `sub.id > 5` and pushed onto the subquery input. Since the inferred predicate must be null-rejecting to be pushed, this drops the subquery's NULL rows and breaks the three-valued `NOT IN` semantics — a NULL in the subquery key must reach the join so the result is empty. Same class of bug as apache#23848, in a different rule. ## What changes are included in this PR? - Skip predicate inference in `infer_join_predicates` when `join.null_aware` is set (mirrors the apache#23848 guard on `FilterNullJoinKeys`). - A `push_down_filter` unit test asserting no predicate is inferred onto the subquery side of a null-aware `LeftAnti` join. - SLT coverage for the failing query, plus a `prefer_hash_join = false` / multi-partition variant. ## Are these changes tested? Yes — new unit test (verified it fails without the guard) and SLT cases. The full optimizer lib suite passes. ## Are there any user-facing changes? No, aside from the correctness fix.
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#23311](apache#23311). - Next in stack: apache#23015. - Extracted from apache#19390. ## Rationale for this change For very small `IN` lists, building or probing a hash table can be more work than just comparing the input value with each constant. For example, for `x IN (10, 20, 30)`, the fast path can behave like: ```text x == 10 OR x == 20 OR x == 30 ``` Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain. “Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic. For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, `Float16` uses `UInt16` storage, `Float32` uses `UInt32` storage, and `TimestampNanosecond` uses `UInt64` storage. `Decimal128` and `IntervalMonthDayNano` use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected. ## What changes are included in this PR? - Adds a const-generic `BranchlessFilter` for small primitive `IN` lists. - Adds thresholds for when this path is used: - up to 16 values for 1-byte types - up to 8 values for 2-byte types - up to 32 values for 4-byte types - up to 16 values for 8-byte types - up to 4 values for 16-byte types - Keeps dispatch concrete and explicit in `strategy.rs`. - Maps each optimized logical type to the comparison representation used by the branchless filter: - `Int8` -> `UInt8` - `Int16`, `Float16` -> `UInt16` - `Int32`, `Float32`, `Date32`, `Time32` -> `UInt32` - `Int64`, `Float64`, `Date64`, `Time64`, `Timestamp`, `Duration` -> `UInt64` - `Decimal128`, `IntervalMonthDayNano` -> their native 16-byte representation - Leaves larger 1-byte and 2-byte lists on the existing bitmap filters. - Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths. - Leaves wider primitive types such as `Decimal256` and unsupported complex types on the generic path. - Keeps the same `IN` / `NOT IN` null behavior as the rest of the stack. - Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run with `release-nonlto`, filtered to the relevant small primitive-list rows: ```bash cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline> ``` Filters used: `narrow_integer`, `primitive/i32/small_list`, `primitive/i64/small_list`, `f32/small_list`, `timestamp_ns/small_list`, and `interval_month_day_nano/small_list`. Method: directly compared Criterion's raw sample minima (`min(time / iterations)`) from `sample.json`. Lower is better; changes within +/-5% are treated as noise. Compared baselines: [apache#23311](apache#23311) -> [apache#23014](apache#23014) Relevant scope: small primitive-list rows. Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%. Largest relevant deltas: | Benchmark | Before | After | Change | |---|---:|---:|---:| | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | <details> <summary>Full relevant table (39 rows)</summary> | Benchmark | Before | After | Change | |---|---:|---:|---:| | `narrow_integer/u8/list=4/match=0%` | 3.86 us | 2.79 us | -27.8% (1.38x faster) | | `narrow_integer/u8/list=4/match=50%` | 3.84 us | 2.78 us | -27.7% (1.38x faster) | | `narrow_integer/u8/list=16/match=0%` | 3.88 us | 3.85 us | -0.8% (within noise) | | `narrow_integer/u8/list=16/match=50%` | 3.84 us | 3.86 us | +0.5% (within noise) | | `narrow_integer/i16/list=4/match=0%` | 3.93 us | 3.18 us | -19.1% (1.24x faster) | | `narrow_integer/i16/list=4/match=50%` | 3.92 us | 3.16 us | -19.5% (1.24x faster) | | `narrow_integer/i16/list=64/match=0%` | 3.96 us | 3.82 us | -3.5% (within noise) | | `narrow_integer/i16/list=64/match=50%` | 3.91 us | 3.80 us | -2.9% (within noise) | | `narrow_integer/i16/list=256/match=0%` | 3.90 us | 3.81 us | -2.5% (within noise) | | `narrow_integer/i16/list=256/match=50%` | 3.97 us | 3.81 us | -4.1% (within noise) | | `narrow_integer/f16/list=4/match=0%` | 3.87 us | 3.16 us | -18.5% (1.23x faster) | | `narrow_integer/f16/list=4/match=50%` | 3.94 us | 3.15 us | -20.2% (1.25x faster) | | `narrow_integer/f16/list=64/match=0%` | 3.87 us | 3.84 us | -0.6% (within noise) | | `narrow_integer/f16/list=64/match=50%` | 3.93 us | 3.85 us | -1.9% (within noise) | | `narrow_integer/f16/list=256/match=0%` | 3.90 us | 3.84 us | -1.5% (within noise) | | `narrow_integer/f16/list=256/match=50%` | 3.87 us | 3.91 us | +1.2% (within noise) | | `nulls/narrow_integer/u8/list=16/match=50%/nulls=20%` | 3.92 us | 4.02 us | +2.5% (within noise) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i32/small_list/list=32/match=0%` | 16.34 us | 13.33 us | -18.5% (1.23x faster) | | `primitive/i32/small_list/list=32/match=50%` | 31.17 us | 13.31 us | -57.3% (2.34x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `primitive/i64/small_list/list=16/match=0%` | 16.34 us | 11.93 us | -27.0% (1.37x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `f32/small_list/list=32/match=0%` | 22.05 us | 13.43 us | -39.1% (1.64x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `timestamp_ns/small_list/list=16/match=0%` | 19.73 us | 12.07 us | -38.8% (1.63x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `interval_month_day_nano/small_list/list=4/match=0%` | 20.12 us | 13.20 us | -34.4% (1.52x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | </details> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Closes apache#23770 - Part of apache#15914 ## Rationale for this change Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out. ## What changes are included in this PR? - Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`. - Register it in `datafusion/spark/src/function/math/mod.rs`. - Enable the `hypot.slt` sqllogictest. The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark. ## Are these changes tested? Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers: - scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13), - double inputs, - NULL propagation when either argument is NULL, - the array path (including a NULL row), - overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`. ## Are there any user-facing changes? Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of apache#23393 . ## Rationale for this change `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only reported the stack size of their `ScalarValue` field plus its heap payload, ignoring the memory held by the underlying `MovingMin` / `MovingMax` sliding-window buffers. For windowed `MIN`/`MAX` over string or list data, the two per-element stacks can hold megabytes of `ScalarValue` payload that the memory pool was never told about, understating accumulator memory usage. ## What changes are included in this PR? - Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and `MovingMax<T>` that reports the two stack buffers' capacity in bytes plus each stored element's heap payload. - Factor the shared implementation into a `moving_stacks_heap_size` free helper so the two types stay in sync. - Include the buffer bytes in `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` via the new method. ## Are these changes tested? Yes. Two new unit tests in `datafusion/functions-aggregate/src/min_max.rs`: - `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only accounting with and without pushed elements. - `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each of the two slots in a `(T, T)` pair contributes independently to the heap payload (mirroring the two independent `Clone`s made by `push`).
## Which issue does this PR close? - Related to apache#21882 (comment). ## Rationale for this change PR apache#21882 adds custom spill file support. Having an example showing how a downstream application can use this new API will help make sure the API is good enough for our needs ## What changes are included in this PR? This PR adds a small example showing how users can back spill files with an `ObjectStore`, using a local object store for a runnable example while keeping the implementation applicable to remote stores. ## Are these changes tested? Yes by CI ## Are there any user-facing changes? Yes. This adds a new example for configuring ObjectStore-backed spill files.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #N/A. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Since DataFusion doesn't typically guartantee wire format compatibility, I remove the backward compatiblity shim that introduced in PR apache#23189 FYI: apache#23189 (comment) ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> DataFusion does not guarantee serialized plans across versions. Keeping `partitioned_by_file_group` therefore leaves a dead schema field and decoder path after `output_partitioning` became the source of truth. Reserve the old field number and name to prevent future reuse. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Yes Signed-off-by: Jiawei Zhao <Phoenix500526@163.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…llable build key (apache#23173) ## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown.
…cimal) (apache#23631) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> I noticed there were some subtle errors with how decimals were handled in scalar values, and also opportunity to remove power calls in favour of precomputed constant tables. Also filling out some other missing support. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> I recommend looking at the commits as they are self contained with detailed messages for each. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…lter pushdown enabled (apache#23638) ## Which issue does this PR close? - Closes apache#23531. ## Rationale for this change `input_file_name()` is `FileSource` dependent like `file_row_index()`, and therefore shouldn't be pushed down into a filter. ## What changes are included in this PR? `PushdownChecker` now handles both UDFs consistently. If we keep adding this sort of metadata functions, we might want a better API to detect them, but for now I think this is a reasonable approach that isn't very invasive. ## Are these changes tested? Additional SLT test that verifies that both function behave correctly with pushdown either enabled or disabled. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
…xpr::check_bigger_cast (apache#23808) (apache#23809) ## Which issue does this PR close? - Closes apache#23808. ## Rationale for this change ref. apache#23807 `CastExpr::check_bigger_cast` is used to determine whether a cast is a widening (order-preserving) conversion. Currently, it classifies `Int32 -> Float32`, `UInt32 -> Float32`, `Int64 -> Float64`, and `UInt64 -> Float64` as widening casts. However, integer-to-float conversions for 32-bit and 64-bit integers lose precision when values exceed the mantissa bit limit (24 bits for `Float32`, 53 bits for `Float64`). For example: `16_777_216_i32 as f32 == 16_777_217_i32 as f32` (both yield 16777216.0f32). Because distinct integer inputs can collapse to the same float output, these casts are not strictly 1-to-1 (injective) and can break suffix sort key ordering in multi-column ordering analysis (e.g. `[CAST(a AS Float32), b]`). ## What changes are included in this PR? - Updated `CastExpr::check_bigger_cast` to exclude precision-losing integer-to-float conversions (`Int32/UInt32 -> Float32` and `Int64/UInt64 -> Float64`). - Added unit test `test_check_bigger_cast_precision_loss` to verify precision-losing casts return `false` while exact conversions (`Int16 -> Float32`, `Int32 -> Float64`, etc.) continue to return `true`. ## Are these changes tested? Yes, new unit test `test_check_bigger_cast_precision_loss` in `cast.rs`. ## Are there any user-facing changes? No breaking API changes. Internal optimizer behavior fix.
…ossible (apache#23702) ## Which issue does this PR close? N/A ## Rationale for this change `SortPreservingMergeStream` is a little complex, so add some guiding comments and make it as textbook-like as possible ## What changes are included in this PR? Added comments, reorder code While this was done this also fixed couple of bugs due to how it work: 1. leftover drain was not counted in the `elapsed_compute` 2. `limit(0)` returns 0 rows and not 1 ## Are these changes tested? Existing tests ## Are there any user-facing changes? Not API ones. `limit(0)` now returns 0 rows
## Which issue does this PR close? - Closes apache#21428. ## Rationale for this change This PR adds `BuildHasher`-based variants for `hash_utils` so callers can compute row hashes with a caller-provided hash builder instead of always using DataFusion's default `RandomState`. The main constraint is performance: `with_hashes` is a hot path, especially for string, dictionary, and nested array hashing. A previous version in apache#21429 caused measurable regressions in the default `RandomState` path, for example `large_utf8: single, no nulls` regressed from roughly `26.7us` to `36.3us`, and `large_utf8: multiple, no nulls` from roughly `112us` to `127us`. This version keeps the default path performance-oriented by avoiding a fully generic `BuildHasher` rewrite of the existing hot loops. ## What changes are included in this PR? This PR adds: - `with_hashes_with_hasher` - `create_hashes_with_hasher` - custom-hasher implementations for primitive, string, binary, byte-view, dictionary, and nested arrays - tests covering custom hashers, multi-column hashing, and dictionary equivalence The implementation intentionally uses a hybrid design: - Default `RandomState` leaf hot paths remain specialized. - Custom `BuildHasher` leaf paths live separately in `hash_utils/build_hasher.rs`. - Nested/structural logic is shared through an internal child-hashing adapter, so struct/list/map/union/run/dictionary behavior does not need to be broadly duplicated. The trade-off is that there is still some duplication for primitive/string/binary leaf loops. That duplication is intentional: those are the hottest loops, and keeping them separate prevents the existing `RandomState` path from becoming generic over `BuildHasher` or being perturbed by the custom-hasher implementation. ## Are these changes tested? Yes. --------- Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…t whether they keep the same ordering of the input (apache#23807) ## Which issue does this PR close? - Closes apache#23798 Related to: - apache#16217 ## Rationale for this change To be able to keep the same sorting order allowing for more optimizations Now comet own `cast` implementation can be recognized as not modifying sort order in the same cases that datafusion cast does. ## What changes are included in this PR? added `strictly_order_preserving` property to `ExprProperties` + varius other places and replaced the hard coded logic for cast (`substitute_cast_ordering`) about keeping input order with more generic approach that now any expression can implement and have the same advantage of sort elimination also marked from_unixtime as keeping ordering to show a case of this optimization ## Are these changes tested? yes ## Are there any user-facing changes? yes, breaking change, added `strictly_order_preserving` property to `ExprProperties` and to `FFI_ExprProperties` this property means that given expression `f` and 2 values from the input column `a` and `b` the following variants are kept: 1. `a.cmp(b) == f(a).cmp(f(b))` 2. nulls maps to nulls Example of satisfying expression: `cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the properties Example of not satisfying: `floor` - floor can not `array_repeat(my_col, 2)` which might look like at first glance as keeping the property as well but in fact it does not. the reason is that `array_repeat(null, 2)` will output list of 2 nulls which breaks the 2nd property that nulls must be kept as nulls how to migrate: Option 1 - keeping the old behavior (safest but least performant) set `strictly_order_preserving` to false Option 2 - Using the new optimization that this opens up: set `strictly_order_preserving` only if the expression keep both variants
## Which issue does this PR close? Part of apache#22330. ## Rationale for this change `ForeignScalarUDF` inherits the default `preserves_lex_ordering`, so producer overrides are lost across the FFI boundary. ## What changes are included in this PR? - Forward `preserves_lex_ordering` through `FFI_ScalarUDF`. - Reuse the existing placement UDF for unit and dynamic-library coverage. ## Are these changes tested? - `cargo test -p datafusion-ffi --features integration-tests` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? The FFI ABI changes. Foreign libraries must rebuild against the new DataFusion version. --------- Signed-off-by: Amogh Ramesh <ramogh2404@gmail.com>
…tonic deques (apache#23826) (apache#23827) ## Which issue does this PR close? - Closes apache#23826 ### What changes are included in this PR? This PR optimizes sliding window `MIN`/`MAX` aggregate functions using a **Sequence-Numbered Monotonic Deque** instead of a Two-Stack Queue. **Revised Design:** - We store `(sequence_number, value)` pairs in a single `VecDeque`, kept in strictly monotonic order. - **`push(val)`**: Evicts dominated elements from the back, then pushes the new value with the current `push_seq` and increments `push_seq`. - **`pop()`**: Increments `pop_seq`. If the front elements sequence number equals the old `pop_seq`, it is expired and popped. - **Benefits**: No secondary FIFO queue (lower memory) and no `clone()` overhead. ### Are these changes tested? Yes, existing tests pass. Added tests for empty-window `pop()` and duplicate-heavy scenarios. ### Are there any user-facing changes? **API Change**: `MovingMin` and `MovingMax` were changed to `pub(crate)` visibility, and their `pop()` methods now return `()` instead of returning a value. Performance is significantly improved (2x-3.5x throughput). --------- Co-authored-by: Pavan <pavan@Lakshmis-MacBook-Air.local>
…ection is live (apache#24355) (apache#24374) Backport of apache#24359 to `branch-55` for the 55.0.0 release, per @timsaucer's request in apache#22393. Stacks cleanly on the already-merged apache#24368 (apache#24354 backport). ## Which issue does this PR close? - Backports the fix for apache#24355 — a second, independent silent wrong-results bug in the same parquet dynamic row-group pruning path as apache#24352. ## Rationale With `pushdown_filters=true` + a TopK dynamic filter, the runtime row-group pruner rebuilds the push decoder via `into_builder().with_row_groups(...)`, which drops row groups **without slicing** the carried flat page-index `RowSelection` to match — a dropped RG's selectors are then applied to the next surviving RG, silently returning wrong rows (no error). The fix declines to build the runtime `RowGroupPruner` when a row selection is present (correctness over the pruning optimization); the proper fix that keeps both is tracked upstream in apache/arrow-rs#10624 / apache#24358. ## Notes - Clean cherry-pick of apache#24359 onto `branch-55` (which now has apache#24354 via apache#24368). No conflicts. - apache#24359 is **approved** on `main` and pending merge; opening this now so it can ride RC3. - Verified locally on this branch: the full `dynamic_row_group_pruning` rust module (9/9) and `dynamic_row_group_pruning.slt` pass; clippy clean. cc @timsaucer @alamb @adriangb
…d/prepend/replace - apache#24365 (apache#24377) This is a back port of apache#24365 into `branch-55`
New commits were added to branch-55 so the changelog is updated.
…ize (apache#24384) (apache#24529) ## Which issue does this PR close? - Part of apache#24462 - Backport of apache#24384 to `branch-55` (for 55.1.0, tracked in apache#24462). - Fixes apache#24383 ## Rationale for this change `UnnestExec` emitted exactly one output batch per input batch, however many rows the unnesting produced, never consulting `datafusion.execution.batch_size`. This let downstream operators receive arbitrarily large batches and made peak memory scale with input batch size times list length instead of `batch_size`. No API change, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#24384 (commit 6187d47). No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage, all tests pass. ## Are there any user-facing changes? `UnnestExec` output batches now respect `datafusion.execution.batch_size` as an upper bound. No API changes. Co-authored-by: Andy Grove <agrove@apache.org>
…s adaptation (apache#24125) (apache#24530) ## Which issue does this PR close? - Part of apache#24462 - Backport of apache#24125 to `branch-55` (for 55.1.0, tracked in apache#24462). - Fixes apache#24109 ## Rationale for this change With `datafusion.execution.parquet.pushdown_filters = true`, a predicate on a struct field was reported as fully handled by the scan whenever the file needed schema adaptation, so `FilterExec` was removed from the plan and the predicate was silently dropped — returning every row instead of the filtered set. This is a correctness bug (wrong results), not specific to 55.0.0, so it fits the backport criteria. ## What changes are included in this PR? Cherry-pick of apache#24125 (commit 40c208e). Git's recursive merge auto-resolved surrounding context differences in `datafusion/physical-expr-adapter/src/schema_rewriter.rs` and `datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt`; no manual conflict resolution or adaptation of the fix itself was required. One follow-up commit adapts a test expectation: `branch-55` doesn't have apache#24130/apache#24315, which taught nested schema pruning to union the leaves needed by mixed whole-column + field-access reads (e.g. `select s, s['y'] from narrow`). Without that optimization, the mixed-access case falls back to reading every physical leaf, so `bytes_scanned` is `219` here instead of the `146` the original PR's test expects on `main`. This is a pre-existing difference in pruning capability, not a correctness regression from this fix. ## Are these changes tested? Yes. Carries the original regression coverage, all tests pass. ## Are there any user-facing changes? `WHERE s['field'] = ...` predicates on struct columns now filter correctly when Parquet filter pushdown requires schema adaptation. No API changes. --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
…te_physical_plan (apache#24690) (apache#24694) This PR is a cherry-pick of apache#24690 onto `branch-55`. See the original PR for a description of the issue. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uction (apache#24723) (apache#24752) This is a back port of apache#24723 onto `branch-55` to support `datafusion-python` upgrade to 55.1.0. The details can be found in the linked PR. This is needed for apache/datafusion-python#1677 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… to planned schema in aggregation (apache#24394) (apache#24699) This is a back port of apache#24394 for `branch-55`. The original issue is apache#24069 Note that the original PR was marked as an auto-detected api change, but I believe that is a false positive. --------- Co-authored-by: Patrick Ribbsaeter <patrickswedish@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… and new_default (apache#24582) (apache#24876) This is a back port of apache#24582 into `branch-55` for inclusion in 55.1.0. Please see the original PR for details. Co-authored-by: Unik Dahal <61407386+unikdahal@users.noreply.github.com>
This is a small update to the Cargo.lock file to update - chacha20 v0.10.0 -> v0.10.2 - h2 v0.4.13 -> v0.4.19 The first was due to a yanked crate. The second was due to a security vulnerability, RUSTSEC-2026-0258. The yanked crate is a warning in cargo audit and will auto-resolve on building to the newer 0.10.2, and the security vulnerability is already updated in `main`.
…asts (apache#23169) (apache#24875) This PR is a backport of apache#23169 onto `branch-55`. I made one update the `use` statement in `planner.rs`. Co-authored-by: Dewey Dunnington <dewey@wherobots.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…ache#24670) (apache#24992) Backports apache#24670 to branch-55. Co-authored-by: Tim Saucer <timsaucer@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR updates the version number and changelog for the 55.1.0 release.
- Relates to apache#22989 but does not close it. More types are coming. - Support the Arrow type `Dictionary` for `approx_distinct` - Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Dictionary` - Tests Yes Yes, `approx_distinct` supports now `Dictionary` but no breaking changes.
…r-defined types (apache#25015) ## Which issue does this PR close? N/A ## Rationale for this change `consume_user_defined_type` maps a Substrait user-defined type straight to a bare `DataType`, so a consumer has no way to mark the resulting `Field` for a downstream reader to recover semantic type lost in that mapping. For example, a UDT representing JSON is commonly mapped to plain `Utf8`, which is then indistinguishable from a real string column once it reaches a consumer that only sees the Arrow schema (e.g. over Arrow Flight, with no access to the original plan). ## What changes are included in this PR? - Adds `consume_user_defined_type_metadata`, a second, additive extension point on `SubstraitConsumer` defaulting to `Ok(None)`. - Applies its result to the `Field` built in `from_substrait_struct_type`, via `Field::with_metadata`. - Existing consumers are unaffected: the default keeps today's behavior exactly (no metadata attached). ## Are these changes tested? Yes, two new unit tests: a consumer that supplies metadata for a `UserDefined` field gets it attached to the field (data type unchanged), and a consumer returning `None` (the default) attaches nothing. ## Are there any user-facing changes? No behavior change for existing consumers. This is a new, optional trait method with a default implementation. (cherry picked from commit d682553)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.