Add a constructor for an all-null column

[emilk]

Related

• Closes Add a constructor for an all-null column #39

What

Column::<Option<L>>::new_null(len) — a column of nulls, built with arrow::array::new_null_array instead of going through the values path and naming a type the column never holds. Same on TypedArray.

ColumnDesc::<Option<L>>::new_null(len) does it from a descriptor, carrying the declared metadata over, so DESC.optional().new_null(n) pads a record batch without repeating the column name.

ColumnDesc::arrow_metadata() single-sources the declared metadata that arrow_field and new_null both need.

Run-end encoding is the one exception: a RunArray has no validity of its own, so Option<Run<K, V>> is unbuildable by any route (from_nullable_values already errors on it). new_null panics there, documented and regression-tested.

Testing

cargo test --all-features. New new_null test covers zero length, nesting, dictionary, fixed-size binary, the TypedArray form, and the descriptor round-trip through a record batch.

Compatibility

Additive.

[emilk]

Post-merge review, as part of reviewing the 0.6 stack for #30. Clippy and tests are clean at the tip of release-0.6. new_null earns its place — from_nullable_values(repeat_n(None::<Vec<u8>>, len)) really does make you name a type the column never holds — and routing it through the descriptor so DESC.optional().new_null(n) doesn't restate the name is what makes it fit the rest of the API.

The run-end panic is length-dependent. Verified at the tip of release-0.6:

Column::<Option<Run<i32, Utf8>>>::new_null(0)   // succeeds, len 0
Column::<Option<Run<i32, Utf8>>>::new_null(2)   // panics

new_null_array for a zero-length RunEndEncoded produces empty run-ends and empty values, so there are no child nulls for Run's downcast to reject and the expect never fires. So the documented "Panics for run-end encoding" holds for every length except the one people pass in edge-case tests. Either check the datatype up front, or narrow the doc to a non-empty run-end column and add the len = 0 case to the test. Related: #37's optional() also reaches Column<Option<Run<K, V>>>, so "which no constructor can build" isn't right either — both notes probably resolve into one paragraph in Run's docs.

The panic is doing the type system's job. This crate otherwise excludes impossible cases at compile time: PrimitiveType gates as_slice, RefType gates Index, InfallibleBuild gates from_values. new_null compiles for every ConcreteType and panics for one. InfallibleBuild is the wrong bound (your test proves Option<Dictionary<i32, Utf8>>::new_null works), but a marker for "this encoding has a top-level validity buffer" moves it to the type system and deletes the expect — the only expect in the crate a user can trigger from safe, obvious code.

Minor: ColumnDesc::new_null goes arrow_metadata() → HashMap → re-collect into BTreeMap; iterating self.metadata directly skips the intermediate. And the two impl<L: ConcreteType> Column<Option<L>> blocks are adjacent with the same bound — typed_array.rs correctly puts new_null in the existing block, so the two files disagree.

— Claude