Skip to content

Preserve list element metadata when rebuilding scalar values - #26092

Open
osipovartem wants to merge 1 commit into
apache:mainfrom
Embucket:fix/list-scalar-field-metadata
Open

osipovartem wants to merge 1 commit into
apache:mainfrom
Embucket:fix/list-scalar-field-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ScalarValue::cast_to reconstructs LIST values through SingleRowListArrayBuilder::with_field. That builder copied the child field name and nullability but dropped its metadata. Casting a LIST to a target field containing nested metadata (for example an Iceberg PARQUET:field_id) therefore produced a scalar with a different type from the requested target, and subsequent batch expansion could fail schema validation.

Copy the field metadata as well. This only runs when constructing a scalar LIST; columnar cast and batch fast paths are unchanged.

Validation

  • New regression test covers LIST scalar cast to a metadata-bearing element field and batch expansion.
  • cargo +1.95.0 test --offline -p datafusion-common --lib: 624 passed.
  • cargo +1.95.0 fmt --all --check: passed.
  • Independent read-only review: approved, no blocking findings.

@github-actions github-actions Bot added the common Related to common crate label Oct 6, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.72%. Comparing base (db83fcc) to head (3acb8ec).

Files with missing lines Patch % Lines
datafusion/common/src/utils/mod.rs 89.47% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26092      +/-   ##
==========================================
- Coverage   82.72%   82.72%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      448179   448194      +15     
  Branches   448179   448194      +15     
==========================================
+ Hits       370754   370757       +3     
- Misses      54895    54902       +7     
- Partials    22530    22535       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants