Skip to content

Move DynColumn to its own file, with private, validated fields - #49

Merged
emilk merged 1 commit into
release-0.6from
emilk/dyn-column-file
Aug 28, 2026
Merged

Move DynColumn to its own file, with private, validated fields#49
emilk merged 1 commit into
release-0.6from
emilk/dyn-column-file

Conversation

@emilk

@emilk emilk commented Aug 28, 2026

Copy link
Copy Markdown
Member

What

  • DynColumn moves out of lib.rs into dyn_column.rs.
  • Its fields are private now, behind field(), array(), and into_parts().
  • DynColumn::try_new validates that the array has exactly the field's datatype, and that a non-nullable field holds no nulls — the two invariants a RecordBatch demands of a column, so a DynColumn can always go into one.

⚠️ Breaking: the struct literal DynColumn { field, array } and the public fields are gone.

Column::into_dyn now takes the field's datatype from the array rather than from L::datatype(). The two agree on everything L pins down, but a validated array can still differ in a detail the logical type does not constrain (the name of a list's inner field), and the field must describe the array exactly.

The two internal callers that build the field and array together (Column::into_dyn, and extraction from a record batch) use a private unvalidated constructor; the derive's extra_columns goes through try_new.

Testing

  • New dyn_column_try_new_validation test: wrong datatype, inner-field nullability, nulls under a non-nullable field, and that the resulting pair fits a RecordBatch.
  • New doctest on try_new.
  • cargo test --all-features and cargo clippy --all-features --all-targets pass.

Compatibility

Breaks source compatibility for anyone constructing a DynColumn by struct literal or reading .field / .array.

`DynColumn`'s fields are now private, behind `field()`, `array()`, and
`into_parts()`. The new `DynColumn::try_new` checks that the array has
exactly the field's datatype, and that a non-nullable field holds no
nulls — the two invariants a `RecordBatch` demands of a column.

`Column::into_dyn` now takes the field's datatype from the array rather
than from `L::datatype()`, so the two halves cannot disagree on a detail
the logical type does not pin down (the name of a list's inner field).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@emilk
emilk marked this pull request as ready for review August 28, 2026 07:04
@emilk
emilk merged commit 4b5460e into release-0.6 Aug 28, 2026
6 checks passed
@emilk emilk mentioned this pull request Aug 28, 2026

@emilk emilk left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge review, as part of reviewing the 0.6 stack for #30. Clippy and tests are clean at the tip of release-0.6. Private fields plus a validating try_new is the right shape, and taking into_dyn's data type from the array rather than L::data_type() is the subtle improvement in here — the field now describes the array exactly, inner field names included.

The derive revalidates extra_columns on every parse, and misattributes the error it can't produce. Generated code went from a struct literal to DynColumn::try_new(…)? per unknown column. Those halves come straight out of a RecordBatch, which arrow has already checked the same two ways, so the error path is unreachable — but it isn't free: array.data_type() != field.data_type() is a recursive comparison per extra column per parse, on the decode hot path. And if it ever did fire, the error carries record_type: "DynColumn" rather than the user's struct name, unlike every other error the derive produces. new_unvalidated is the right call there (it is exactly the "field and array built together, already validated" case the doc describes), or map the error to #record_type.

expected: format!("{:?}", field.data_type()) is the noisy formatter. Verified: DataType implements Display, so a List<i64> mismatch reads

expected List(Field { data_type: Int64, nullable: true })   // {:?}
expected List(Int64)                                       // {}

The derive does the same at quiver.rs:688, and ErrorKind::WrongDataType's own format string uses {actual:?}, so the wider fix is one line in error.rs plus these two call sites.

— Claude

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant