Move DynColumn to its own file, with private, validated fields - #49
Conversation
`DynColumn`'s fields are now private, behind `field()`, `array()`, and `into_parts()`. The new `DynColumn::try_new` checks that the array has exactly the field's datatype, and that a non-nullable field holds no nulls — the two invariants a `RecordBatch` demands of a column. `Column::into_dyn` now takes the field's datatype from the array rather than from `L::datatype()`, so the two halves cannot disagree on a detail the logical type does not pin down (the name of a list's inner field). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
emilk
left a comment
There was a problem hiding this comment.
Post-merge review, as part of reviewing the 0.6 stack for #30. Clippy and tests are clean at the tip of release-0.6. Private fields plus a validating try_new is the right shape, and taking into_dyn's data type from the array rather than L::data_type() is the subtle improvement in here — the field now describes the array exactly, inner field names included.
The derive revalidates extra_columns on every parse, and misattributes the error it can't produce. Generated code went from a struct literal to DynColumn::try_new(…)? per unknown column. Those halves come straight out of a RecordBatch, which arrow has already checked the same two ways, so the error path is unreachable — but it isn't free: array.data_type() != field.data_type() is a recursive comparison per extra column per parse, on the decode hot path. And if it ever did fire, the error carries record_type: "DynColumn" rather than the user's struct name, unlike every other error the derive produces. new_unvalidated is the right call there (it is exactly the "field and array built together, already validated" case the doc describes), or map the error to #record_type.
expected: format!("{:?}", field.data_type()) is the noisy formatter. Verified: DataType implements Display, so a List<i64> mismatch reads
expected List(Field { data_type: Int64, nullable: true }) // {:?}
expected List(Int64) // {}
The derive does the same at quiver.rs:688, and ErrorKind::WrongDataType's own format string uses {actual:?}, so the wider fix is one line in error.rs plus these two call sites.
— Claude
What
DynColumnmoves out oflib.rsintodyn_column.rs.field(),array(), andinto_parts().DynColumn::try_newvalidates that the array has exactly the field's datatype, and that a non-nullable field holds no nulls — the two invariants aRecordBatchdemands of a column, so aDynColumncan always go into one.DynColumn { field, array }and the public fields are gone.Column::into_dynnow takes the field's datatype from the array rather than fromL::datatype(). The two agree on everythingLpins down, but a validated array can still differ in a detail the logical type does not constrain (the name of a list's inner field), and the field must describe the array exactly.The two internal callers that build the field and array together (
Column::into_dyn, and extraction from a record batch) use a private unvalidated constructor; the derive'sextra_columnsgoes throughtry_new.Testing
dyn_column_try_new_validationtest: wrong datatype, inner-field nullability, nulls under a non-nullable field, and that the resulting pair fits aRecordBatch.try_new.cargo test --all-featuresandcargo clippy --all-features --all-targetspass.Compatibility
Breaks source compatibility for anyone constructing a
DynColumnby struct literal or reading.field/.array.