You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1871883
Browse filesBrowse the repository at this point in the historyBrowse files
fix: preserve nulls when unnesting structs (#25389)
## Which issue does this PR close?
Closes#25293.
## Rationale for this change
Unnesting a NULL struct can expose its stored child values. A
non-nullable child field can also make a valid query fail. Both cases
now produce NULL values.
## What changes are included in this PR?
Output fields include the parent's nullability. UNNEST uses the
parent-null helper shared with field extraction, including its null-free
fast path. The helper retains value buffers where the array type
supports a validity bitmap and keeps the union/run-end fallback.
The operator benchmark covers integer and 128-byte string fields across
batch sizes and null densities.
## What is the testing strategy for this PR?
### Testing Done
SQL regressions cover schema nullability. One physical-operator
regression checks values and buffer identity together across null-free,
mixed-null and all-null parents, sliced and nested arrays, and encoded
children.
Run from the repository root:
```sh
cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \
-c "CREATE TABLE base AS SELECT named_struct('a', 1, 'b', 'x') AS s; CREATE TABLE masked AS SELECT nullif(s, s) AS s FROM base; SELECT unnest(s) FROM masked;"
cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \
-c "SELECT unnest(arrow_cast(NULL, 'Struct(\"a\": non-null Int32)'));"
```
The JSON writer omits NULL-valued fields. Before the fix, the first
query exposes values and the second fails:
```text
[{"masked.s.a":1,"masked.s.b":"x"}]
Error: Arrow error: Invalid argument error: Column '__unnest_placeholder(arrow_cast(NULL,Utf8("Struct("a": non-null Int32)"))).a' is declared as non-nullable but contains null values
```
With the fix, both queries return:
```text
[{}]
[{}]
```
The SQL rerun used a cached CLI built from source matching this PR. The
full local workspace run stopped during linking with `ld: write()
failed, errno=28 (No space left on device)`, before tests started.
#### Operator measurement
The recorded comparison of [the shared-helper
implementation](9fffb0a)
against [the main
control](d477a2b)
used one ARM64 host and `release-nonlto`. For 65,536 rows of 128-byte
strings with 50% null parents, the control took about 3.05 us and the
helper took 4.15 us. Main omits the parent mask and returns incorrect
null results, so this measures the added correctness cost, not a
speedup.
<details>
<summary>Benchmark commands and raw output</summary>
From this PR's checkout, prepare the main control with only the
benchmark and its manifest entry:
```bash
root=$(mktemp -d)
before=d477a2b038730647bef258cb98e58978b0c74699
after=9fffb0a72273cef82b7c48091a257ac1bf77684c
bench=datafusion/physical-plan/benches/unnest.rs
manifest=datafusion/physical-plan/Cargo.toml
git worktree add --detach "$root/before" "$before"
git worktree add --detach "$root/after" "$after"
git diff "$before" "$after" -- "$manifest" "$bench" |
git -C "$root/before" apply
export CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$root/target"
```
The measured commands use identical inputs and options:
```bash
(cd "$root/before" &&
cargo bench --locked --profile release-nonlto -p datafusion-physical-plan --bench unnest -- \
'utf8_128_bytes_nulls_50/65536' --noplot --sample-size 30 \
--warm-up-time 1 --measurement-time 2 --save-baseline current-main)
(cd "$root/after" &&
cargo bench --locked --profile release-nonlto -p datafusion-physical-plan --bench unnest -- \
'utf8_128_bytes_nulls_50/65536' --noplot --sample-size 30 \
--warm-up-time 1 --measurement-time 2 --baseline current-main)
```
Main control:
```text
unnest_struct/utf8_128_bytes_nulls_50/65536
time: [3.0444 µs 3.0483 µs 3.0536 µs]
thrpt: [21.462 Gelem/s 21.499 Gelem/s 21.527 Gelem/s]
```
Candidate:
```text
unnest_struct/utf8_128_bytes_nulls_50/65536
time: [4.1454 µs 4.1524 µs 4.1638 µs]
thrpt: [15.740 Gelem/s 15.783 Gelem/s 15.809 Gelem/s]
change:
time: [+36.598% +37.187% +37.828%] (p = 0.00 < 0.05)
thrpt: [−27.446% −27.107% −26.792%]
Performance has regressed.
```
These are operator measurements, not end-to-end query timings. No
null-free performance claim is made.
</details>
## Are there any user-facing changes?
NULL structs now unnest into NULL fields rather than stored values or an
error.
---------
Signed-off-by: 1fanwang <1fannnw@gmail.com>
0 commit comments