Add optional() / required() to ColumnDesc and Column

[emilk]

Related

• Closes Add ColumnDesc::optional() for reading a declared-non-nullable column tolerantly #35

What

optional() widens a column to nullable, required() narrows it back:

Self::COLUMN_CHUNK_KEY.optional().extract(&batch)?;   // reads the nulls
Self::COLUMN_CHUNK_KEY.optional().arrow_field();      // declared nullable
column.optional();                                    // Column<T>         -> Column<Option<T>>
column.try_required()?;                               // Column<Option<T>> -> Column<T>

A column can be declared non-nullable and still legitimately hold nulls on some code path (e.g. concatenating a batch that has the column with one that does not). Previously that needed a second descriptor restating the name and dropping the metadata.

Both directions are idempotent: LogicalType gains Optional and Required associated types, so .optional().optional() does not pile up Option<Option<T>>. They are bound to the same Typed — nullability lives in the validity bitmap, not in the downcast representation — so the conversions copy nothing and re-downcast nothing. Column::try_required is the only fallible one, erroring with UnexpectedNulls.

⚠️ Breaking for hand-written LogicalType impls: they must now state type Optional = Option<Self>; and type Required = Self;. Associated-type defaults are unstable, so there is no way to make this additive. The derive and the type macros need no change.

Testing

column_desc_optional_and_required and column_optional_and_required: metadata and name carry over, only field nullability flips; both directions idempotent; nulls rejected only by extract / try_required; empty and all-null columns as the boundary cases. Plus four doctests.

Compatibility

Additive for users of the derive and the built-in logical types. Breaking for hand-written LogicalType impls, as above.

[emilk]

Post-merge review, as part of reviewing the 0.6 stack for #30. Clippy and tests are clean at the tip of release-0.6. The core insight — Optional/Required share one Typed, so the conversion is a pure retype with nothing to re-validate — is what makes this a feature rather than a wrapper around try_new, and spelling the idempotence assertions as type annotations in the test is a nice trick.

Column::<Run<K, V>>::optional() builds a type #42 documents as unbuildable. Verified at the tip of release-0.6:

Column::<Run<i32, Utf8>>::try_from_values(["a", "a", "b"]).unwrap().optional().to_vec()
// -> [Some("a"), Some("a"), Some("b")]

Column<Option<Run<K, V>>> is reachable, and reads as all-Some forever because a RunArray has no top-level validity buffer for Option<L>::is_null to consult. Not unsound, and optional() being generic over LogicalType is by design — but #42's "unbuildable by any route" is wrong, and a user reaching for .optional() specifically to tolerate nulls gets silence instead of an error. A sentence in Run's docs is probably the whole fix.

TypedArray doesn't get the feature. into_optional / try_into_required already exist as pub(crate) and already do the whole job. TypedArray is pitched (since #23) as "the same array with the same value API, minus the metadata"; nullability isn't metadata, so this is the first place that promise doesn't hold. It's a rename away.

Minor: nothing stops a hand-written impl writing type Optional = Self; on a non-nullable type, silently making optional() a no-op. type Optional: LogicalType<Typed = Self::Typed, Required = Self::Required> (and the mirror) holds for every impl in the crate — the only question is whether the solver copes with the mutual reference.

— Claude