Repository navigation
Conversation
## Which issue does this PR close? Part of apache#25148. ## Rationale for this change Five feature-check jobs currently run without a Rust dependency cache. Reusing compatible compilation artifacts should reduce repeated dependency compilation and CI runner usage while preserving the existing feature checks and parallel jobs. ## What changes are included in this PR? - Add dependency caching to the `datafusion-common`, `datafusion-proto`, `datafusion-ffi`, `datafusion-functions`, and `datafusion-spark` feature-check jobs using the existing `amd-ci-check` key. - Let the functions check save the cache on `main` to populate GitHub's cache backend. The workspace check already saves to the separate Runs-On cache backend. The other four jobs only restore caches. - Clarify the cache comments. Job names, runner choices, feature-check commands, and required checks remain unchanged. ## What is the testing strategy for this PR? - Compared the parsed workflows before and after the change: removing the five new cache steps produces the original configuration, including all 26 jobs and their commands. - `python3 ci/scripts/check_asf_yaml_status_checks.py` passed for all 35 required checks. - `ci/scripts/check_no_cargo_install_in_workflows.sh`, targeted spelling checks, and `git diff --check` passed. - Actionlint passed for the complete cache producer and consumer job definitions. Full-workflow actionlint still reports existing `parallel` blocks and a numeric service-port lookup outside the changed jobs. This changes workflow configuration only. Cache hit rates and elapsed-time savings need measurement in hosted CI with populated caches. A cache miss still compiles dependencies and runs the existing checks. ## Are there any user-facing changes? No.
…ache#25228) ## Which issue does this PR close? Closes apache#25226. ## Rationale for this change MIN/MAX queries can return incorrect results when a Parquet file contains a mix of row groups with and without bounds. A missing bound is converted to a NULL in the per-row-group statistics array. Aggregating that array skips the NULL, allowing another group's endpoint to be reported as an exact bound for the entire file. The optimizer can then answer the aggregate from incomplete statistics. For example, a file contains `["a", "b"]` with statistics and two 8192-character strings starting with `z` without statistics. `SELECT LENGTH(MIN(a)), LENGTH(MAX(a))` previously returned `1, 1`; the correct result is `1, 8192`. ## What changes are included in this PR? - Invalidate a file-level bound when any row group has no converted bound and may contain non-NULL values. Return `Precision::Absent` so that incomplete statistics cannot establish the file's extrema. - Handle minimum and maximum independently, preserving a usable bound when only the other endpoint is missing. - Preserve bounds from other groups when the group with no bound is empty or is proven all NULL. The all-NULL check requires both the column value count and the explicitly recorded NULL count to equal the row count. - Express the active-accumulator precondition with `expect`: callers already check for `Some`, and the path that clears the accumulator returns immediately. This removes an unreachable fallback return. ## What is the testing strategy for this PR? `test_min_max_require_complete_row_group_bounds` covers absent statistics, independently missing minimum/maximum values, unknown and partial NULL counts, empty groups, all-NULL groups, and complete bounds. Every case runs in both row-group orders. `parquet_missing_bounds.slt` verifies both the stored string lengths and the aggregate results. Its fixture is generated at runtime using public Rust Parquet writer APIs, enabling chunk statistics for the first row group and disabling them for the second. Disabling statistics is deliberate: parquet-rs truncates long extrema rather than omitting them. No generated Parquet file is committed and no footer is manually edited. Local validation: - All 10 tests in `metadata::tests::statistics_tests` passed. - `parquet_missing_bounds.slt` passed with the fix. Running the same fixture and SQL test against unmodified main reproduced the incorrect `1, 1` result. - LLVM coverage confirmed that the removed fallback return had zero executions and that the replacement precondition assertion was exercised 36 times by the statistics tests. The updated Codecov patch percentage remains subject to the next CI run. ## Are there any user-facing changes? MIN/MAX queries return correct results for Parquet files with incomplete row-group bounds. Valid statistics remain usable for each independently complete endpoint, including when other groups are empty or all NULL. There are no public API changes. --------- Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
## Which issue does this PR close? - N/A ## Rationale for this change `bit_xor(DISTINCT c)` with `GROUP BY` fails with `column types must match schema types, expected List(Int16) but found Int16`: `groups_accumulator_supported` ignored `is_distinct`, so the plan got the non-distinct groups accumulator while `state_fields` declared the `List` state. ## What changes are included in this PR? `groups_accumulator_supported` now returns false only for `bit_xor(DISTINCT)`, matching `accumulator` and `state_fields`. `bit_and` and `bit_or` are idempotent, so they keep the vectorized path. ## What is the testing strategy for this PR? Three `aggregate.slt` queries: the two shapes `SingleDistinctToGroupBy` does not rewrite (a non-distinct aggregate alongside, a `FILTER`) fail on `main`; a `bit_and`/`bit_or` `DISTINCT` row pins the narrower predicate. ## Are there any user-facing changes? `bit_xor(DISTINCT)` with `GROUP BY` now works. No API change. --------- Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
## Which issue does this PR close? - Part of apache#21048 and apache#24487. Both issues stay open. ## Rationale for this change The three extended test suites run only in the merge queue, on release branches, and by manual dispatch. Their `cargo test` commands and environment variables exist only inside `extended.yml`. A developer who wants to reproduce a merge-queue failure must copy them by hand. This PR defines each command once in `xtask`, so local runs and GitHub Actions use the same definition. ## What changes are included in this PR? - `cargo xtask ci step test` gets three variants: `extended`, `hash-collisions`, and `sqlite`. Each variant reproduces the current CI invocation and sets only the environment that its suite needs. The `sqlite` variant sets no `RUST_BACKTRACE`, because its CI job skips the builder setup on purpose. - `extended.yml` calls the three commands. Triggers, job IDs, job names, runners, containers, the clean working tree check, and `cargo clean` do not change. - `xtask/README.md` documents the commands, their environment, prerequisites, and the parts of a CI job that a command does not reproduce. `--explain` prints the underlying command without running the suite: ```sh cargo xtask ci step test extended --explain ``` ## What is the testing strategy for this PR? - Snapshot tests in `xtask/src/ci_steps.rs` cover the exact command, working directory, and environment of each variant, plus rejection of an unknown variant. - `check_asf_yaml_status_checks.py` passes. All required check names are unchanged. - Each new command ran the real suite locally through `xtask` on macOS. The `extended`, `hash-collisions`, and `sqlite` suites all passed with a clean working tree. The extended jobs skip ordinary pull request updates, so green PR checks do not exercise these commands. The merge queue does. ## Are there any user-facing changes? No. The GitHub CI behavior is the same. Developers get three new local commands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…e#24988) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> For UDFs using `Numeric` signature with more than one argument (currently mod & pmod from datafusion-spark), they cannot handle inputs `Null, Int` whereas they accept `Int, Null`. Fixing the coercion logic to ensure both are accepted. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Modify numeric type coercion handling to ignore null types ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Added unit & SLT tests ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No
## Which issue does this PR close? - Closes apache#25071 ## Rationale for this change With `enable_unions_to_filter` enabled, `UNION DISTINCT` branches containing different computed projections below their filters can be incorrectly treated as equivalent. This produces wrong result values and row counts without an error. ## What changes are included in this PR? `unions_to_filter` now skips the rewrite when a projection appears below a branch filter. Alias nodes remain safe to strip. A regression test covers branches computing `a + 100` and `a + 200` below their filters. ## What is the testing strategy for this PR? - `cargo +1.97.0-x86_64-pc-windows-gnullvm test --locked -p datafusion-optimizer unions_to_filter --lib` (9 passed) - The standalone reproducer returns `[1100, 1200]` with the optimizer setting both disabled and enabled after the fix. - `git diff --check` ## Are there any user-facing changes? This prevents incorrect query results when the opt-in `unions_to_filter` optimizer rule is enabled.
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#24102](apache#24102). ## Rationale for this change Arrow represents every `Utf8View` and `BinaryView` value with a 16-byte view. When a value is at most 12 bytes long, the complete value is stored directly in that view: ```text 32-bit length | up to 12 value bytes, zero-padded ``` Equal short values therefore have the same 128-bit view. DataFusion can look up that view directly, without reading another buffer or using Arrow's general value comparison. This PR uses that faster path when every non-null value in the `IN` list is at most 12 bytes long. The byte-view adapter treats each inline view as a `Decimal128` key and passes it to the shared primitive selector from [apache#24283](apache#24283), the same selector used by apache#24102: - Lists with up to four non-null values use the direct-comparison filter from apache#23014. - Lists with 5 or more use the shared 128-bit hash-set filter. `Decimal128` is only a 16-byte container here; no decimal operations are performed. A longer input value cannot match a short list value because its encoded length is different, so it can be rejected without reading its backing bytes. If the list itself contains a value longer than 12 bytes, DataFusion keeps using the general filter. Null list entries do not affect this choice, but are still recorded for SQL null behavior. After unwrapping any dictionary, the input and list must use the same view type: `Utf8View` is not mixed with `BinaryView`, and regular `Utf8` and `Binary` keep their existing paths. ## What changes are included in this PR? - Adds the short-value path for `Utf8View` and `BinaryView`. - Passes the 128-bit keys to the shared primitive selector instead of making a separate filter choice. - Keeps the general filter for longer list values or mismatched types. - Preserves slices, dictionaries, nulls, `IN`, and `NOT IN`. ## Are these changes tested? Tests cover both view types and both faster paths; the 4/5-value and 12/13-byte boundaries; long inputs and lists that require the general filter; exact-type routing; slices and dictionaries; input and list nulls; and `IN` and `NOT IN`. ## Are there any user-facing changes? No. SQL results and public APIs are unchanged. ## Automated benchmark snapshot [Automated run](apache#24088 (comment)) on an aarch64 Neoverse V2 runner, comparing [apache#24102](apache#24102) with this PR at exact commit `00dd083e`. The benchmark evaluates 8,192-row batches after filter construction. Lower is better. All 15 reported rows improved, with a 35.9% geometric-mean time reduction. | Benchmark | Before | After | Change | |---|---:|---:|---:| | `utf8view/short_8b/list=4/match=0%` | 18.7 us | 11.6 us | -38.0% (1.61x faster) | | `utf8view/short_8b/list=4/match=50%` | 39.3 us | 11.6 us | -70.5% (3.39x faster) | | `utf8view/short_8b/list=16/match=0%` | 18.9 us | 14.1 us | -25.4% (1.34x faster) | | `utf8view/short_8b/list=16/match=50%` | 39.6 us | 30.1 us | -24.0% (1.32x faster) | | `utf8view/short_8b/list=64/match=0%` | 18.6 us | 13.4 us | -28.0% (1.39x faster) | | `utf8view/short_8b/list=64/match=50%` | 38.5 us | 28.4 us | -26.2% (1.36x faster) | | `utf8view/short_8b/list=256/match=0%` | 18.5 us | 13.4 us | -27.6% (1.38x faster) | | `utf8view/short_8b/list=256/match=50%` | 38.3 us | 25.7 us | -32.9% (1.49x faster) | | `utf8view/len_12b/list=16/match=0%` | 18.6 us | 15.4 us | -17.2% (1.21x faster) | | `utf8view/len_12b/list=16/match=50%` | 40.0 us | 18.3 us | -54.2% (2.19x faster) | | `utf8view/len_12b/list=64/match=0%` | 18.7 us | 13.7 us | -26.7% (1.36x faster) | | `utf8view/len_12b/list=64/match=50%` | 39.6 us | 25.9 us | -34.6% (1.53x faster) | | `nulls/utf8view/short_8b/list=16/match=50%/nulls=20%` | 38.7 us | 25.8 us | -33.3% (1.50x faster) | | `nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN` | 38.5 us | 24.7 us | -35.8% (1.56x faster) | | `nulls/utf8view/short_8b/list=16/match=50%/nulls=50%` | 29.2 us | 17.8 us | -39.0% (1.64x faster) | --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Closes apache#24462 55.1.0 has now been released. This PR brings in the changelog and version number to `main`. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? Closes apache#5332. ## Rationale for this change Selecting a column that doesn't exist lists every field in the schema. On a 200 column table that's about 2.8k characters, and the useful part of the message - the name that wasn't found and the "Did you mean" suggestion - ends up scrolled off the top. ``` > SELECT not_a_column FROM wide; Error: Schema error: No field named not_a_column. Valid fields are wide.col_0, wide.col_1, ... wide.col_199. ``` Filed in 2023, still reproduces. ## What changes are included in this PR? Cap the list at 20 fields and summarise the rest: ``` Valid fields are wide.col_0, ... wide.col_19 and 180 others. ``` 2.8k characters down to 356. Schemas at or under 20 fields print the same as before, so the existing message assertions in dfschema.rs, column.rs, expr_rewriter and the dataframe tests are untouched - they're all well under that. The issue suggested either dropping the list like Postgres does or truncating it. I kept it and bounded it, since the list is useful on narrow schemas and closest_valid_field already handles the common typo. Happy to switch to the Postgres behaviour instead if you'd rather. ## Are these changes tested? Four unit tests in datafusion/common/src/error.rs: a narrow schema, exactly 20 fields, 21 fields (checks it says "1 other" not "1 others"), and 200 fields. cargo test -p datafusion-common: 608 passed. datafusion-expr --lib: 258 passed. cargo fmt clean. Two things in my checkout that aren't from this change and reproduce on unmodified main: the arrow_test_data doctest fails because the testing/ submodule isn't initialised, and clippy reports an unused import of crate::config::TableParquetOptions. ## Are there any user-facing changes? The wording of SchemaError::FieldNotFound when a schema has more than 20 fields. No API change. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> Part of apache#24727 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> Reduce binary code size by reducing monomorphized code. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> Extracted non-generic default method body implementations of `ScalarUDFImpl`: - `coerce_types` - `schema_name` - `return_field_from_args` - `output_ordering` Measurements: `cargo llvm-lines --release -p datafusion-expr --lib | head -3` Before (1d9a982): ``` Lines Copies Function name ----- ------ ------------- 1247760 26213 (TOTAL) ``` After (7278ffd4fe7f): ``` Lines Copies Function name ----- ------ ------------- 1247097 26207 (TOTAL) ``` `cargo build --profile release-nonlto --bin datafusion-cli && stat -f%z ../shared-cargo-target/release-nonlto/datafusion-cli` Before (1d9a982): ``` 186946400 ``` After (7278ffd4fe7f): ``` 186574320 ``` So 663 fewer LLVM lines, 372 080 fewer bytes (non-lto build of datafusion-cli) ## What is the testing strategy for this PR? Just code-shuffling, no behaviour change. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? Slightly binary size, and maybe faster compilation. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…Stream (apache#25158) ## Which issue does this PR close? - Closes apache#24980 ## Rationale for this change `SingleHashAggregateStream` ignores the distinct soft limit pushed into `AggregateExec`. As a result, single-stage `SELECT DISTINCT ... LIMIT n` queries consume all input even after enough distinct groups have been collected. This change stops input consumption once the in-memory hash table contains at least the requested number of distinct groups. Any existing spills are merged before producing output, and the downstream limit operator enforces the exact output row count. If spilling has already occurred, the optimization is skipped to keep the spill and replay path simple. In that case, the stream consumes the remaining input and follows the existing spill merge path. ## What changes are included in this PR? * Add a distinct soft-limit check to `SingleHashAggregateStream` after processing each input batch. - Stop consuming input when the soft limit is reached before spilling. * Reuse the input-exhausted transition when the soft limit is reached, preserving spill merging and output preparation. * Document how the optimization applies to single-stage distinct aggregation. - Update the `AggregateExec::limit_options` documentation to list `SingleHash` as supporting distinct soft limits. - Add an end-to-end SQL test that verifies: - the query is planned with a single-stage `AggregateExec`; - the distinct soft limit is pushed into the aggregate; - the aggregate stops consuming its input after reaching the soft limit; - the query returns the correct number of rows. ## Are these changes tested? Added unit tests covering: * Reaching the soft limit without spilling. * Reaching the soft limit after spilling, preserving spilled groups and deduplicating overlapping groups. * Rejecting further input consumption after the soft limit is reached in the spill case. Following test commands have been executed and passed - `cargo test --profile=ci --test sqllogictests` - `cargo test -p datafusion` - `cargo test -p datafusion-cli` ## Are there any user-facing changes? Single-stage `DISTINCT` queries with a limit can stop consuming input earlier, reducing unnecessary work. Query semantics are unchanged.
…#25236) ## Which issue does this PR close? - Closes apache#25235. ## Rationale for this change Calling schema_name to string is expensive ## What changes are included in this PR? Uses a hashset of names to prevent reformatting the name ``` logical_unnest_plus_200_columns time: [1.0161 ms 1.0183 ms 1.0205 ms] change: [−79.252% −79.058% −78.900%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe logical_unnest_plus_1000_columns time: [14.751 ms 14.799 ms 14.849 ms] change: [−86.220% −86.145% −86.079%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low mild 6 (6.00%) high mild 2 (2.00%) high severe logical_unnest_in_200_exprs time: [3.1697 ms 3.1815 ms 3.1953 ms] change: [−71.260% −71.088% −70.910%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe ``` ## What is the testing strategy for this PR? - Added benchmarks - Added tests directed to `DedupedProjection` - Added a sql logic test for a case @xudong963 identified in my initial fix ## Are there any user-facing changes? None
## Which issue does this PR close? - Closes apache#24970. ## Rationale for this change `INSERT` currently aligns source expressions with target columns by storage `DataType` only. If both fields use the same storage type but the target is an Arrow extension type, the cast is skipped and the DML projection loses the target extension metadata. A table sink can then receive a schema that does not describe the target logical type. ## What changes are included in this PR? After storage-type coercion, compare the resulting field metadata with the target table field. When they differ, use the existing field-aware `Cast` representation to enforce the target metadata. Existing casts are replaced rather than nested. The regression uses an unannotated `FixedSizeBinary(16)` source and an `arrow.uuid` target so the behavior is independent of any particular downstream consumer. Existing parameterized UUID INSERT snapshots are updated to show the target field-aware cast. ## What is the testing strategy for this PR? - Added `plan_insert_preserves_target_extension_metadata`. - Ran all 575 `datafusion-sql` integration tests. - Ran `cargo clippy -p datafusion-sql --all-targets --all-features -- -D warnings`. - Ran `cargo fmt --all -- --check`. ## Are there any user-facing changes? Yes. SQL `INSERT` plans now preserve target Arrow field metadata, including extension metadata. There are no public API changes.
## Which issue does this PR close? Related to apache#18083. Full support for NULL partition values remains covered by that issue. ## Rationale for this change Partitioned writes read partition values without checking their validity. A NULL integer partition key can be written under `p=0`, so reading the output turns NULL into zero and silently changes the data. For example: ```sql CREATE TABLE src(p INT, v INT) AS VALUES (NULL, 1), (0, 2); COPY src TO '/tmp/null_partition/' STORED AS PARQUET PARTITIONED BY (p); ``` Both rows can end up in the zero partition. This PR rejects the write with `NULL values are not supported for partition column 'p'`. ## What changes are included in this PR? - Check logical nulls before extracting partition keys, including NULL dictionary values referenced by valid dictionary indices. - Continue accepting nullable columns when the selected rows contain no NULL values. - Add focused unit tests and a `copy.slt` regression covering rejection and a filtered successful write. ## What is the testing strategy for this PR? - The rejection regressions fail without the production fix and pass with it. - Focused partition-key tests: 5 passed with `RUST_BACKTRACE=1`. - `cargo clippy --all-targets --all-features -- -D warnings` passed. - The extended workspace Rust suites passed after excluding four host memory-measurement runners that also fail on current main in this environment. - The SQL logic harness passed all 513 files, including encrypted Parquet coverage. - `cargo fmt --all -- --check` and `git diff --check` passed. ## Are there any user-facing changes? Partitioned writes containing NULL partition keys now return an execution error. Callers can filter or replace these values before writing. Validation happens per batch, so files written by earlier batches can remain after an error.
## Which issue does this PR close? Related to apache#24258. This covers runtime operands, floating-point literals, and volatile expressions in compound IN-list rewrites. The NULL-semantics changes in that PR remain separate. ## Rationale for this change The optimizer performs set intersection and difference using expression equality. Different expressions can evaluate to the same value, so these rewrites can remove matching rows even when every column is non-nullable: ```sql CREATE TABLE in_list_values(x INT NOT NULL, a INT NOT NULL, b INT NOT NULL) AS VALUES (1, 1, 1); SELECT x FROM in_list_values WHERE x IN (a, 2, 10, 11) AND x IN (b, 5, 12, 13); ``` The query should return `1`; the rewrite produces an empty intersection and returns no rows. Structural comparison also distinguishes positive and negative floating-point zero. Deduplicating volatile operands can change how many times they are evaluated. ## What changes are included in this PR? - Restrict intersection and difference rewrites to non-null, non-floating literals. - Guard compound rewrites against volatile inputs and list items. - Preserve valid union rewrites for nonvolatile runtime expressions. - Add unit regressions for runtime values, signed zero, mixed integer coercion, and volatility, plus runtime-value cases in `predicates.slt`. ## What is the testing strategy for this PR? - The new regressions fail without the production fix and pass with it. - Focused optimizer tests: 6 passed. - `cargo clippy --all-targets --all-features -- -D warnings` passed. - The extended workspace Rust suites passed after excluding four host memory-measurement runners that also fail on current main in this environment. - The SQL logic harness passed all 513 files, including encrypted Parquet coverage. - `cargo fmt --all -- --check` and `git diff --check` passed. ## Are there any user-facing changes? Affected compound IN predicates retain matching rows and preserve volatile expression evaluations. No public API changes. ## Downsides Queries excluded by the new guards may perform more comparisons because the optimizer retains their original predicates.
## Which issue does this PR close? No linked issue. The regression below reproduces the problem on current upstream. ## Rationale for this change Grouped CORR uses raw sums whose subtraction loses precision when values have a large common offset. It can return a negative correlation for a column correlated with itself: ```sql SELECT g, corr(x, x) FROM (VALUES (1, 1000000000.0), (1, 1000000007.0), (1, 1000000015.0), (2, 1.0) ) AS t(g, x) GROUP BY g; ``` Group 1 can return `-1` instead of `1`. ## What changes are included in this PR? - Use paired Welford updates and centered moments for grouped correlation. - Merge partial states using weighted mean differences, applying weights before multiplying deltas to avoid avoidable intermediate overflow. - Match the scalar CORR state layout across update, merge, singleton conversion, and preserving/draining emission. - Add regressions for large offsets, multiple batch sizes, scalar/grouped state interchange, and large-range merges, plus a grouped self-correlation case in `aggregate.slt`. ## What is the testing strategy for this PR? - The large-offset and scalar/grouped state compatibility regressions fail without the production fix and pass with it. - Focused correlation tests: 7 passed. - `cargo clippy --all-targets --all-features -- -D warnings` passed. - The extended workspace Rust suites passed after excluding four host memory-measurement runners that also fail on current main in this environment. - The SQL logic harness passed all 513 files, including encrypted Parquet coverage. - `cargo fmt --all -- --check` and `git diff --check` passed. - H2O groupby q09 returned the same 6,320,797 rows on every run. Across 20 release-mode runs, current main averaged 479.5 ms and this PR averaged 373.7 ms. Excluding each process's first run, the median was 333.9 ms on main and 312.7 ms on this PR, so the stability fix showed no regression in this environment. ## Are there any user-facing changes? Grouped CORR produces stable results for the covered large-offset inputs. Floating-point results can differ in their final bits. SQL signatures and output types remain unchanged. ## Downsides Centered updates add arithmetic, including per-row divisions. The grouped intermediate state representation changes from raw sums to centered moments, so partial states produced by the old implementation cannot be mixed with the new representation.
## Which issue does this PR close? Closes a planner gap where `ORDER BY ALL` only accepted raw column projections and was silently ignored for set-operation outputs. ## Rationale for this change `ORDER BY ALL` means sorting by every output column in select-list order. Expanding it to the equivalent 1-based ordinal keys supports computed expressions, aliases, aggregate outputs, wildcard-expanded projections, and set-operation outputs through one planner path. The ordinal expansion also sorts already-projected columns rather than evaluating computed expressions again. Physical execution remains on DataFusion's existing vectorized `SortExec`; this adds no row-wise conversion or custom physical operator. ## What changes are included? - expand `OrderByKind::All` to ordinal `OrderByExpr`s from the output width - provide output width for non-`SELECT` set expressions instead of dropping `ORDER BY ALL` - add focused execution coverage for computed expressions, null-order options, set operations, and aggregate output ## Are these changes tested? - `cargo +1.95.0 test -p datafusion-sqllogictest --test sqllogictests -- order_by_all.slt --test-threads 1` - `cargo +1.95.0 test -p datafusion-sql --test sql_integration -- --test-threads 8` (591 passed) - `cargo +1.95.0 clippy -p datafusion-sql --all-targets -- -D warnings` - `cargo +1.95.0 fmt --all -- --check` Snowflake dialect parsing support is proposed independently in apache/datafusion-sqlparser-rs#2502; this planner change is generic to every dialect that already emits `OrderByKind::All`.
## Which issue does this PR close? - Closes apache#23920 ## Rationale for this change just more uses of the new property and comment on some non strictly order preserving for why not ## What changes are included in this PR? marked if something is order preserving or not, for those who are not I've added a comment why not, since some expressions might be confusing why not ## Are these changes tested? no ## Are there any user-facing changes? not api ones
## Which issue does this PR close? - Part of apache#21048. The issue stays open for the other checklist items. ## Rationale for this change `dev/rust_lint.sh` is the local mirror of the CI lint jobs, but it does not run the Markdown link check. CI runs `ci/scripts/markdown_link_check.sh` in `dev.yml`. A developer can push a broken internal link after a clean local lint run and then see the failure only in CI. This PR adds the existing checker to the local suite. ## What changes are included in this PR? - `dev/rust_lint.sh` runs `ci/scripts/markdown_link_check.sh` as a read-only step, after the workflow install check and before the Rust documentation build. The `--write` and `--allow-dirty` flags never reach it. - The runner loads `LYCHEE_VERSION` from `ci/scripts/utils/tool_versions.sh` and installs that version when `lychee` is missing. An installed `lychee` is used as is, which is the runner's existing policy for other tools. - `ci/scripts/markdown_link_check.sh` becomes executable, because the runner invokes each registered script directly. The script body, `lychee.toml`, the file selection, and the GitHub workflow do not change. - `docs/source/contributor-guide/testing.md` documents the new behavior and keeps the standalone instructions. ## What is the testing strategy for this PR? - A disposable fixture with stubbed steps and tools: the checker runs once with no arguments in check mode and in both write modes, a missing `lychee` triggers exactly the pinned install command, a present `lychee` triggers no install, and a checker failure stops the suite before the later steps. - A disposable clone with an injected broken internal link: the standalone checker and `./dev/rust_lint.sh` both fail with the same exit code and name the file. After the fix, the full suite passes. - The real checker and the full `./dev/rust_lint.sh` pass on this branch with `lychee` 0.23.0. ## Are there any user-facing changes? No. Developers get the link check in the local lint suite. CI is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Which issue does this PR close? Closes apache#25239. Follows up on apache#21907 and [Arrow apache#6485](apache/arrow-rs#6485). ## Rationale for this change An absent Parquet `null_count` is unknown. Treating it as zero can discard NULL rows during `NULLS FIRST` TopK and `IS NULL` pruning, and let metadata-only `COUNT(column)` return an incorrect result. ## What changes are included in this PR? Preserve missing null counts in the shared row-group statistics adapter and file statistics. Remove the per-caller compatibility flag so static pruning, runtime pruning, and fully matched proofs use the same interpretation. Add an upgrade guide covering affected writers, plan changes, and migration. ## What is the testing strategy for this PR? Regressions cover missing and explicit-zero counts, static pruning, file-statistics precision, `COUNT`, and TopK with ASC/DESC, NULLS FIRST/LAST, and dynamic pruning enabled/disabled. The affected cases fail without the fix. Extended workspace tests, strict all-target/all-feature Clippy, and the full lint suite pass. ## Are there any user-facing changes? Correct results for Parquet files with missing null counts. Affected files can lose `IS NULL` pruning, metadata-only `COUNT(column)`, and `NULLS FIRST` TopK pruning. Sort pushdown can also retain a full `SortExec` for `ORDER BY` on nullable columns, even when files are sorted and non-overlapping. This includes older files written with parquet-rs before 53.1.0, used by DataFusion releases before 42.1.0. The DataFusion 56 upgrade guide explains how to identify and rewrite affected files. Explicit counts retain their existing behavior. No public API changes. --------- Signed-off-by: peterxcli <peterxcli@gmail.com>
## Which issue does this PR close? - N/A ## Rationale for this change Periodic update to latest stable Rust toolchain. ## What changes are included in this PR? * Bump pinned toolchain * Bump `depcheck` sub-workspace toolchain * Update a few references in docs * Remove `from_iter_instead_of_collect` from workspace clippy allow list (lint was removed in 1.98) * Resolve new clippy warnings that appear with clippy 1.98 ## What is the testing strategy for this PR? Compiles cleanly, all clippy / lint checks pass with no new warnings. ## Are there any user-facing changes? No.
…chain (apache#25286) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> `datafusion::physical_plan::joins::chain` is a public module with no public member. See https://docs.rs/datafusion/latest/datafusion/physical_plan/joins/chain/index.html I think it was made public by mistake, this PR revert this module to private. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#24318 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> See issue ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No.
## Which issue does this PR close? - Part of apache#318. - Umbrella PR: apache#23738. - All required ASOF feature layers (apache#23828 through apache#23832) are merged. ## Rationale for this change This is the final core benchmark layer of the ASOF JOIN stack. It uses the standard SQL benchmark framework so the workloads share the existing runner, result format, and Criterion mode. ## What changes are included in this PR? - Add an `asof_join` SQL benchmark suite, runnable with `./bench.sh run asof_join` or `benchmark_runner`. - Add seven workloads covering relative input sizes, available ordering, equality-group cardinality and skew, match direction, and payload width. - Describe each workload and its explicit ASOF match semantics in the query. - Generate pre-sorted Parquet inputs under `DATA_DIR/asof_join` with `./bench.sh data asof_join`. The benchmark load hook only registers them with `WITH ORDER`, keeping data generation and input sorting outside the measured query. - Assert that every workload plans to `AsOfJoinExec`. ## Are these changes tested? Yes: - `cargo fmt --all` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-benchmarks --lib --bins` - `DATA_DIR=/tmp/asof_join CARGO_COMMAND='cargo run' ./bench.sh data asof_join` - `DATA_DIR=/tmp/asof_join CARGO_COMMAND='cargo run' ./bench.sh run asof_join 7` - `cargo run -p datafusion-benchmarks --bin benchmark_runner -- asof_join --iterations 1 --partitions 4 --path /tmp/asof_join` - `cargo run -p datafusion-benchmarks --bin benchmark_runner -- asof_join --query 7 --partitions 4 --path /tmp/asof_join --criterion` The pre-sorted workload's physical plan contains `AsOfJoinExec` without an input `SortExec`. These smoke runs validate the suite and are not presented as performance claims. ## Are there any user-facing changes? This adds ASOF workloads to the repository benchmark tooling. It does not change query semantics or runtime behavior. This is the final core item tracked by apache#23738. It does not depend on the optional floating-point follow-up apache#24375.
…pache#25193) - ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields. - For some Parquet schemas, where we request a view transform for a nested field, this is translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data on that nested field, the actual predicate execution needs a cast, because the schemas are different - the predicate refers to a string view, but the actual data schema refers to a string ## Which issue does this PR close? - Closes apache#25192 ## Rationale for this change - Rationale is explained in apache#25192 - Also, I don't see any reason (explicit comments) why this function would not work on deeply nested schemas ## What changes are included in this PR? - Modification of the `apply_file_schema_type_coercions` function to work on nested fields - Respective tests for the functionality ## What is the testing strategy for this PR? - All tests pass - Newly added unit tests pass ## Are there any user-facing changes? - Unsure ? Probably not ? The documentation in the function does NOT explicitly say the current behavior.
## Which issue does this PR close? - Closes apache#23022. ## Rationale for this change Predicate subqueries such as `EXISTS`, `IN`, and `NOT IN` are valid scalar expressions in a `SELECT` list, but DataFusion currently only decorrelates them when they appear in filters. Projection queries therefore reach physical planning with an unsupported logical subquery expression. This continues the work from the now-closed stale apache#23039 and covers multiple and correlated projection subqueries as well as SQL null semantics. ## What changes are included in this PR? - decorrelate `EXISTS`, `NOT EXISTS`, `IN`, and `NOT IN` expressions inside projections into mark joins - materialize projected `IN`/`NOT IN` values with SQL three-valued logic by tracking matches, nulls, and empty subqueries - preserve projection output names and avoid duplicate columns in correlated subquery projections ## What is the testing strategy for this PR? Optimizer plan tests cover projected `IN` rewriting. New SQLLogicTest cases cover uncorrelated and correlated `EXISTS`/`IN`, multiple subqueries in one projection, empty inputs, null inputs, and `NOT IN` three-valued logic. Locally validated with: - `cargo test -p datafusion-optimizer decorrelate_predicate_subquery` (53 passed) - `cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection` - `cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` ## Are there any user-facing changes? Yes. Predicate subqueries can now be evaluated as projected scalar expressions. There are no public API changes.
## Which issue does this PR close? Fixes the [security audit failure](https://github.com/apache/datafusion/actions/runs/34914826729/job/104210077720) caused by [RUSTSEC-2026-0285](https://rustsec.org/advisories/RUSTSEC-2026-0285.html). ## Rationale for this change The locked rustls 0.23.39 accepts TLS 1.3 handshake messages across encryption level boundaries and now fails the security audit. Rustls 0.23.45 fixes the vulnerability. ## What changes are included in this PR? Update rustls to 0.23.45 in Cargo.lock, together with its required aws-lc-rs, aws-lc-sys, and rustls-webpki dependency updates. ## What is the testing strategy for this PR? The existing security audit reproduces the vulnerability with the original lockfile and passes with the updated lockfile: ```sh cargo audit --ignore RUSTSEC-2026-0194 --ignore RUSTSEC-2026-0195 ``` The workspace extended tests also pass: ```sh RUST_BACKTRACE=1 cargo test --profile ci \ --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \ --workspace --lib --tests --bins \ --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption ``` Also passed `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `./dev/rust_lint.sh`. ## Are there any user-facing changes? Builds using the checked-in lockfile use the patched TLS implementation. No DataFusion API changes.
…ng (apache#25261) ## Which issue does this PR close? - Closes apache#25260. ## Rationale for this change A malformed `null_regex` panics the query task instead of returning an error. `CsvFormat::infer_schema_from_stream` compiled the pattern with ```rust let regex = Regex::new(null_regex.as_str()) .expect("Unable to parse CSV null regex."); ``` so any pattern the `regex` crate rejects aborted the task. It is reachable straight from SQL, through a `CREATE EXTERNAL TABLE` that leaves its columns to schema inference: ```sql CREATE EXTERNAL TABLE t STORED AS CSV LOCATION 'data.csv' OPTIONS ('format.has_header' 'true', 'format.null_regex' '('); ``` ``` task 9 panicked with message "Unable to parse CSV null regex.: Syntax( regex parse error: ( ^ error: unclosed group )" ``` An invalid regex is a bad option value, not an internal invariant, so it should come back as an error naming the pattern. ## What changes are included in this PR? The regex is compiled once, before the per-chunk loop, and a failure is propagated with `exec_datafusion_err!` rather than panicking. Hoisting it also stops the pattern being recompiled for every chunk of the inference stream. The error text matches the one used on the read side in apache#25254, so the same bad option reads the same whichever path hits it first. ## What is the testing strategy for this PR? A `statement error Unable to parse CSV null regex` case in `datafusion/sqllogictest/test_files/csv_files.slt`. I checked it is not vacuous: with this change reverted, that case fails with the panic quoted above rather than an error, so the test reproduces the bug. `cargo fmt --check`, `cargo clippy --all-targets` and the crate's unit tests are clean. ## Are there any user-facing changes? An invalid `null_regex` now produces an error and leaves the session usable, where it previously panicked the task. No API changes. Independent of apache#25254 — that one is the read path in `source.rs`, this is the inference path in `file_format.rs` — but they touch the same crate, so whichever lands second may want a trivial rebase. Co-authored-by: Prajwal Narayana <sprajwalln@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Which issue does this PR close? None; found by sweeping the scalar functions with edge-case arguments. ## Rationale for this change `array_resize` with a negative size raises an internal error, which tells the caller they hit a DataFusion bug and asks them to open a report: ```console > SELECT array_resize(make_array(1,2,3), -1, 0); Internal error: array_resize: failed to convert size to usize. This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues ``` The size is an argument to the query, so a negative value is the caller's input rather than a broken internal invariant. Sending someone to the issue tracker for their own argument is misleading, and it hides what was actually wrong. The behaviour was already meant to be an error. `array_resize.slt` has expected this call to fail since before this change, but with a bare `query error`, so the internal error satisfied it and went unnoticed. ## What changes are included in this PR? Reject a negative size with an execution error naming the value, which matches how the neighbouring maximum-size check already reports bad input: ```console > SELECT array_resize(make_array(1,2,3), -1, 0); Execution error: array_resize: size must not be negative, got -1 ``` Valid input is untouched: `0` still returns `[]`, `5` still grows to `[1, 2, 3, 9, 9]`, and a NULL size still returns NULL. ## Are these changes tested? Yes. The slt file gains the negative case for both `List` and `LargeList`, plus `i64::MIN` for the boundary, each pinned to the new message so a regression back to an internal error fails the suite. ```console $ cargo test -p datafusion-sqllogictest --test sqllogictests -- array Progress: 53/53 files completed (100%) $ cargo test -p datafusion-functions-nested --lib resize test result: ok. 4 passed; 0 failed ``` The pre-existing `query error` case for `-5` still passes, so the contract that a negative size fails is unchanged. ## Are there any user-facing changes? The error for a negative size changes from an internal error to an execution error with a clear message. No valid call changes behaviour, and no API changes. --------- Signed-off-by: 1fanwang <1fannnw@gmail.com> Co-authored-by: Neil Conway <neil.conway@gmail.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…4979) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> `PartialSortStream` concatenates each incoming batch with the buffered incomplete prefix before looking for a completed prefix boundary. When one prefix spans multiple batches, this repeatedly copies rows accumulated from earlier batches. When a batch contains multiple completed prefix groups, the completed region is also sorted using the full ordering even though the prefix ordering is already satisfied. This adds unnecessary copying and sorting work, especially for wide batches and workloads containing multiple prefix groups per batch. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> - Keep the unfinished trailing prefix as a list of `RecordBatch` slices instead of concatenating it with every incoming batch. - Detect completed prefix groups within each batch and compare prefix expression results directly across batch boundaries. - Sort completed groups only by the remaining suffix ordering. - Concatenate a single completed prefix at most once. - For multiple completed prefixes, evaluate suffix expressions once per source batch and materialize the final output with one interleave operation. - Preserve fetch handling, input release, empty-batch handling, and zero-column batch semantics. ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> The implementation is covered by the existing `PartialSortExec` unit tests and the additional SQL logic test cases in `group_by.slt`. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No. This is an internal performance improvement. ## Benchmark ``` group main optimized ----- ---- --------- partial_sort/rows_per_prefix/100 3.10 12.1±0.13ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec partial_sort/rows_per_prefix/1000 4.17 13.2±0.07ms ? ?/sec 1.00 3.2±0.14ms ? ?/sec partial_sort/rows_per_prefix/10000 1.30 2.8±0.02ms ? ?/sec 1.00 2.1±0.07ms ? ?/sec partial_sort/rows_per_prefix/20000 1.41 3.0±0.22ms ? ?/sec 1.00 2.1±0.15ms ? ?/sec partial_sort/rows_per_prefix/5000 3.85 12.4±0.27ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec partial_sort/rows_per_prefix/8192 1.56 2.8±0.05ms ? ?/sec 1.00 1780.8±34.70µs ? ?/sec ```
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 1530 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy review recommended (neutral) Detected 7 PR taxonomy bucket(s): Security Evidence, Harness Drift, Install Manifest Integrity, CI/CD Recommendation, Cost/Token Risk, PR Review/Salvage Evidence, Agent Config Review. Scanned 1530 changed file(s). Roadmap taxonomy buckets: Security EvidenceSecurity-sensitive changes should carry explicit scanner, code-scanning, or focused regression evidence. Signals:
Paths:
Harness DriftHarness-facing changes can drift across Claude Code, Codex, OpenCode, and shared adapter surfaces. Signals:
Paths:
Install Manifest IntegrityInstall manifests, plugin metadata, and shipped skills should stay synchronized with user-facing setup guidance. Signals:
Paths:
CI/CD RecommendationCI, dependency, coverage, and contract signals should be routed into follow-up checks or verification work. Signals:
Paths:
Cost/Token RiskAI routing, usage, and token-budget changes should include budget or usage-limit evidence. Signals:
Paths:
PR Review/Salvage EvidencePR review, stale-closure, and triage automation should carry stale-closure, reviewer-thread, and reopen-flow evidence so useful closed work can still be salvaged. Signals:
Paths:
Agent Config ReviewAgent, command, skill, MCP, and local instruction changes should be reviewed as executable agent configuration. Signals:
Paths:
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 2/7 areas (29%) across 1530 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 1530 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Config AuditCommit: No changed-config issues detected (success) Scanned 12 config file(s) present at this commit across 12 changed config path(s) and found no issues in the supported security rules. Changed config files:
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Harness AuditCommit: Harness warnings detected (neutral) Scanned 12 changed config file(s) and found 1 harness issue(s).
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
github-actions
Bot
added
documentation
Owner
Author
|
Closing this incorrectly targeted PR: the fork's main branch is stale. I will submit the one-line change against apache/datafusion main, where its diff is one line. |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n\nThe docs deployment uses
sync --delete, which removes manually published �ersions/ directories because they are not part of the current development build. Exclude /versions/ from deletion so released documentation survives normal development-site deployments.\n\nThis is intentionally separate from apache#24792 so the deployment behavior can be reviewed independently.