Release 0.6.0 by emilk · Pull Request #30 · rerun-io/quiver

emilk · 2026-08-27T15:09:58Z

What

Bumps the version to 0.6.0 and opens the release branch. Keep the list above updated as PRs are retargeted or merged.

Compatibility

Source-breaking, see the individual PRs:

quiver::Error becomes breaking to destructure (Box ErrorKind inside Error #31)
as_slice() on a primitive newtype changes element type (Column<T>::as_slice() now yields &[T] for newtypes #41), and column[i] on one now yields the newtype rather than the representation (Borrow the newtype, not the representation, on primitive newtypes #56)
DynColumn's fields are private (Move DynColumn to its own file, with private, validated fields #49)
datatype → data_type throughout (Rename Datatype/datatype to DataType/data_type #50)
every Column constructor takes a name (Give Column a name #52), and Column's == now compares the name too
ColumnDesc's declared metadata compares as a map rather than as an ordered slice (Follow-ups from the 0.6 post-merge reviews #55)

🤖 Generated with Claude Code

Opens the 0.6 release branch. `CHANGELOG.md` is generated from merged PR titles at release time, so it is not touched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Enable the pedantic clippy lints we had opted out of Removes the pedantic opt-out block, keeping `float_cmp` and `redundant_closure_for_method_calls` on 'allow': * `float_cmp`: the only hits are exact roundtrip checks of exactly representable values in tests. `float_cmp_const` is still on. * `redundant_closure_for_method_calls`: it wants `std::iter::Iterator::collect::<Vec<_>>` in place of `|inner| inner.collect::<Vec<_>>()`, which is worse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Avoid a sign-losing cast in `AnyList` `FixedSizeListArray::value_length` returns `i32`. Use the `ArrowNativeType::as_usize` conversion already used for `value_offset` on the neighboring line instead of an `as` cast. Fixes `clippy::cast_sign_loss`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Mark `ColumnDesc::new`, `DynColumnDesc::new` and `ColumnError::for_column` as `#[must_use]` Fixes `clippy::must_use_candidate`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Compute ns/element from `as_secs_f64` in the `sum` bench Avoids the `u128 as f64` cast entirely. The remaining `usize as f64` is a printed timing, so it gets a local `#[expect]`. Fixes `clippy::cast_precision_loss`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add `ColumnDesc::to_dyn` for dropping to a `DynColumnDesc` Also `impl From<&ColumnDesc<C>> for DynColumnDesc`. The declared metadata is dropped, since `DynColumnDesc` does not carry any. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add `Column::into_dyn` for converting to a `DynColumn` The arrow field takes its name from the argument, its datatype and nullability from the logical type, and its metadata from the column. Zero-copy: the array is moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add `DynColumn::try_into_column`, the inverse of `Column::into_dyn` Validates the array against the logical type and downcasts it (zero-copy), carrying over the arrow field metadata. Errors name the field, like the other by-name extractors do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test the new `DynColumn`/`DynColumnDesc` conversions Covers metadata round-tripping, `Option<…>` as the only source of field nullability, nested inner-field nullability, and that validation follows the array's nulls rather than the field's nullable flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Name the `Column::into_dyn` parameter `column_name` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix the broken intra-doc link to `DynColumn::try_into_column` `DynColumn` is not in scope in `column.rs`, so the link needs the `crate::` path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the now-redundant explicit link target on `Field` `Field` is imported since `Column::into_dyn` uses it, so the explicit `arrow::datatypes::Field` target makes `rustdoc::redundant_explicit_links` fire under `cargo doc --document-private-items`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make `TypedArray` public `TypedArray<L>` is the data half of a `Column<L>`: the validated, downcast arrow array without the per-column metadata. It was crate-internal, so an array that isn't a record batch column had to be wrapped in a `Column` with an empty metadata map. The value API now lives on `TypedArray` and `Column` delegates to it: - the iterators moved over, and are renamed `TypedArrayIter` / `TypedArrayIntoIter` (they are what `Column::iter` returns too) - constructors (`from_values`, `from_nullable_values`, `Default`, `From<Vec>`, `FromIterator`), `iter`, `to_vec`, `slice`, `Index`, and `as_slice` - `Column::as_typed_array` / `into_typed_array`, and `From<TypedArray>` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add `ColumnDesc::typed_array` Validates a loose `ArrayRef` against the column's logical type, without needing to name that type, and labels errors with the column and record type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test and document the standalone `TypedArray` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Keep `ColumnIter` / `ColumnIntoIter` as deprecated aliases The rename to `TypedArrayIter` / `TypedArrayIntoIter` no longer breaks downstream code: the old names stay as deprecated type aliases, so this can go out in a minor release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Link the main types from the crate docs, and to each other The crate-level docs (the README for `quiver`, the module docs for `quiver_types`) now open with `Column`, `TypedArray`, and `ColumnDesc`, and each of the three explains its relationship to the other two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix redundant explicit link targets in `ColumnDesc` docs `cargo doc --document-private-items` resolves `[`DynColumn`]` on its own, so the explicit `(crate::DynColumn)` target is an error under `-D warnings`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Document that `ColumnDesc` works standalone The docs read as if a descriptor only ever came out of `#[derive(Quiver)]`. `ColumnDesc::new` is a `const fn`, so you can declare your own; say so, with a doctest, and stop describing the fields as derive-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the now-redundant explicit link target on `Field` `Column::into_dyn` imports `Field`, so the explicit `arrow::datatypes::Field` target makes `rustdoc::redundant_explicit_links` fire under `cargo doc --document-private-items`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * `ColumnDesc<L>` instead of `ColumnDesc<Column<L>>` (#24) * `ColumnDesc<L>`, not `ColumnDesc<Column<L>>` The descriptor now names the logical type, which reads better and is the honest parameter: a descriptor yields both a `Column<L>` (`extract`) and a `TypedArray<L>` (`typed_array`), so the wrapper type was never the thing it was parameterized by. The derive used to paste the field type verbatim; it now pulls `L` out of the `Column<L>` path segment, and says so when the parameter is missing. ⚠️ Breaking: annotations spell `ColumnDesc<Utf8>` instead of `ColumnDesc<Column<Utf8>>`. No transitional alias is possible — `impl<L: LogicalType> ColumnDesc<L>` and `impl<L> ColumnDesc<Column<L>>` overlap as far as coherence can tell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test and document `ColumnDesc<L>` Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Build `Column<L>` on demand in the derive `ColumnKind::Wrapper` keeps only the logical type; the three sites that want the column type spell `#krate::Column<#logical_type>`, which also makes the generated code independent of how the field's type was written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Split `ColumnDesc::new` and `new_with_metadata` Most columns declare no metadata, so `&[]` was noise at every call site. The derive picks the short constructor unless the field declares metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Stop mentioning the old `ColumnDesc<Column<L>>` spelling Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Match `ColumnKind::Wrapper` exhaustively Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop a redundant comment in the `ColumnDesc<L>` test Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Box `ErrorKind` inside `Error` `Error` was 96 bytes, driven by the two three-field `ErrorKind` variants (`WrongDatatype` and `WrongArrayType`, 72 bytes each). It rides in the `Err` arm of every `Result` in the crate, so boxing the kind takes it to 24 bytes; the allocation only happens on the cold error path. `Error::new` boxes for you, so no construction site has to name `Box`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Build boxed errors from the derive-generated code Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Match on `*err.kind` in the tests A `Box<ErrorKind>` cannot be pattern-matched through, so the assertions that used to spell out the whole `Err(Error { .. })` pattern now check `record_type` separately and match the dereferenced kind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Assert at compile time that `Error` stays small Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…criptors (#32) * Give `ColumnDesc` and `DynColumnDesc` the usual derives `DynColumnDesc` derives `Clone, Copy, Debug, Eq, PartialEq`. `ColumnDesc` gets them hand-written: `#[derive]` would put each trait's bound on `L`, but a descriptor holds no `L` — only a `PhantomData<fn() -> L>`, which is `Copy`, `Eq`, and `Debug` whatever `L` is. `Debug` skips the `PhantomData`. `to_dyn` now takes `self` by value, since `Self: Copy`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test the `ColumnDesc` / `DynColumnDesc` derives Uses a `ColumnDesc<Utf8>` for the `Debug` and `Eq` checks: `Utf8` itself derives nothing, so those lines only compile because the impls put no bound on `L`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add `ColumnDesc::optional()` for reading a declared-non-nullable column tolerantly A column can be declared non-nullable and still legitimately hold nulls on some code path, e.g. after concatenating a batch that has the column with one that does not. `optional()` gives a `ColumnDesc<Option<L>>` that reads and declares such a column, carrying the record type, name, and metadata over, so the name stays single-sourced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make `optional()` idempotent, and add the `required()` inverse `LogicalType` gains two associated types: `Optional` (`Option<Self>`, but `Self` for `Option<L>`) and `Required` (`Self`, but `L::Required` for `Option<L>`). Both are bound to the same `Typed`, since nullability lives in the validity bitmap and not in the downcast representation, so converting between the two spellings is free. `ColumnDesc::optional()` now returns `ColumnDesc<L::Optional>` rather than nesting into `Option<Option<L>>`, and gains `required()`. `Column` gets the same pair: `optional()` is infallible, `try_required()` errors with `UnexpectedNulls` on a column that really does hold nulls. Both carry the metadata over, and neither copies or re-downcasts. Breaking for hand-written `LogicalType` impls, which must now state `type Optional` and `type Required` (associated-type defaults are unstable). The derive and the type macros need no change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add `nullable` / `non_nullable` doc aliases The methods are named after the Rust type the caller gets (`Option<L>`), not after arrow's field flag — and a `nullable()` next to the existing `NULLABLE` bool const would read like a getter for it. The aliases let a rustdoc search in arrow's vocabulary find them anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

`Deref<Target = [L::Native]>` and `AsRef<[L::Native]>` for the columns that already have `as_slice` (primitive and fixed-size binary, non-nullable), so a `Column<u64>` is a drop-in for the `ScalarBuffer<u64>` it replaces and `&self.chunk_byte_sizes` keeps compiling. Method resolution keeps `Column`'s own methods: an inherent method wins over a dereferenced one, so `iter`, `len`, `get`, and `to_vec` still read the logical values. Indexing stays with the existing `Index<usize>` impl, so range slicing needs an explicit `&(*column)[a..b]` — documented, and covered by the test. Same pair on `TypedArray`. Closes #40 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add a constructor for an all-null column `Column::<Option<L>>::new_null(len)` (and the `TypedArray` form) build a column of nulls directly, with `arrow::array::new_null_array`. Before, that went through the values path and had to name a type the column never holds: Column::<Option<Binary>>::from_nullable_values( std::iter::repeat_n(None::<Vec<u8>>, num_rows), ) `ColumnDesc::<Option<L>>::new_null(len)` does the same from a descriptor, carrying the declared metadata over, so `DESC.optional().new_null(n)` pads a record batch that is missing the column without repeating its name. Run-end encoding is the one exception: a `RunArray` has no validity of its own, so `Option<Run<K, V>>` is unbuildable by any route. Documented, with a regression test. Closes #39 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add `ColumnDesc::arrow_metadata` Single-sources the declared metadata that `arrow_field` and `new_null` both need, and lets a caller stamp it on a field they build themselves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Depend on and re-export `bytemuck` Zero-dependency, and re-exported as `quiver::bytemuck` so that macro users do not have to depend on it themselves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let a `primitive` newtype set `PrimitiveType::Native = Self` `newtype_datatype!(ChunkId, FixedSizeBinary<16>, primitive)` used to forward `Native` to the representation, so `Column<ChunkId>::as_slice()` handed back `&[[u8; 16]]` — the raw representation the newtype exists to hide — and every caller re-cast it by hand. The `primitive` arm now sets `Native = Self` and does the cast once, inside quiver, with `bytemuck::must_cast_slice`. That requires the newtype to be `Pod`, and checks size and alignment at compile time, so a newtype that is not layout-compatible fails to build rather than silently misreading. A newtype that cannot be `Pod` uses the new `primitive(raw)` arm, which keeps the old behavior. The built-in `NonZero*` and `char` impls use it. Closes #36 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the `primitive(raw)` arm: two forms, not three A newtype that cannot be `Pod` now simply has no bulk read from the macro. Where handing back the representation's values is still useful, the `PrimitiveType` impl is three lines by hand — shown in the docs, and what the built-in `NonZero*` and `char` columns now use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Re-export the `bytemuck` derive, pinning `bytemuck_derive` to 1.11 1.12 moved to `syn` 3, which duplicates the `syn` 2 that `quiver_derive` uses and fails `cargo deny check bans`; `bytemuck` itself only asks for ^1.10.2, so the lock pin is enough. A newtype then reaches `Pod` with `#[derive(quiver::bytemuck::Pod)]` plus `#[bytemuck(crate = "::quiver::bytemuck")]`, which points the derive back at the re-export — no `unsafe`, and no direct `bytemuck` dependency. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop a redundant explicit link target `cargo doc --document-private-items` runs with `-D warnings`, and `rustdoc::redundant_explicit_links` fires on a label that already resolves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

`Column` and `TypedArray` already report their arrow datatype statically; a descriptor could only do so via `arrow_field()`. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

`DynColumn`'s fields are now private, behind `field()`, `array()`, and `into_parts()`. The new `DynColumn::try_new` checks that the array has exactly the field's datatype, and that a non-nullable field holds no nulls — the two invariants a `RecordBatch` demands of a column. `Column::into_dyn` now takes the field's datatype from the array rather than from `L::datatype()`, so the two halves cannot disagree on a detail the logical type does not pin down (the name of a list's inner field). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Match arrow, which spells it `arrow::datatypes::DataType`. Renamed: * `ConcreteType::datatype()`, `Column::datatype()`, `TypedArray::datatype()`, `ColumnDesc::datatype()` -> `data_type()` * `ErrorKind::WrongDatatype`, `ColumnError::WrongDatatype` -> `WrongDataType` * `newtype_datatype!` -> `newtype_data_type!`, `try_newtype_datatype!` -> `try_newtype_data_type!` * `src/datatype.rs` -> `src/data_type.rs` Prose says "data type", two words. `.typos.toml` now enforces this: the word `datatype` is a typo for "data type", which CI already checks. The plural is untouched — it is the name of the `arrow::datatypes` module. `CHANGELOG.md` is excluded: it quotes old PR titles verbatim. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

`ColumnDesc` and `DynColumnDesc` already have a public `name` field; `DynColumn` has one on its arrow field. A method on all three means a column's name is reached the same way whichever of them you hold. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: make `TypedArray::optional` / `try_required` public `Column` has had both; the data half needs them too, now that tests and docs that do not care about a column name use a `TypedArray`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat!: give `Column` a name A column of a record batch has a name and metadata; a bare array has neither. `Column` already carried the metadata, but not the name — which is why `into_dyn` had to ask for one, and why `Column::name()` could not exist. Every `Column` constructor now takes the name: Column::<Utf8>::from_values("sensor", ["kitchen"]) Column::try_new("sensor", array)? Column::new("sensor", typed_array) // names a `TypedArray` Column::<Option<Binary>>::new_null("chunk_key", 3) Column::<i64>::empty("frame") The impls that cannot supply a name are gone from `Column`, and stay on `TypedArray`: `Default`, `From<Vec<T>>`, `FromIterator<T>`, `TryFrom<ArrayRef>`, `From<TypedArray<L>>` (use `Column::new`), and the `TryFrom<Vec<T>>` impls for `Dictionary` and `Run`. `Column::into_dyn` no longer takes a name — it uses the column's own. Rename on the way out with `with_name`. `PartialEq` and `Debug` include the name. `ColumnDesc` grows `new_from_values` / `try_new_from_values`, which take the name and declared metadata from the descriptor, so a `#[derive(Quiver)]` struct literal does not repeat the field name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: use `TypedArray` where a column name is beside the point Most of the standalone tests, the benchmarks, the soundness tests, and the logical-type doctests exercise element access, iteration, slicing, and encodings — none of which involve a name or metadata. Those now use `TypedArray`; a `Column` is left only where the name, the metadata, or the record batch itself is what is under test. The docstrings say the distinction outright: a column has a name and metadata, an array has neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: tighten the README wording on column names Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: split the standalone tests by type, and rename to match `tests/column.rs` had become 80% `TypedArray` tests, under names that still said "column". The 52 that do not involve a name or metadata move to `tests/typed_array.rs`, and are renamed accordingly (`binary_columns` -> `binary_arrays`, `standalone_flat_column` -> `flat_array`, …), local variables included. The soundness tests and the benchmarks get the same treatment. `tests/column.rs` is left with the 13 tests that are actually about a column: the name, the metadata, and the record batch boundary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…try_new` (#53) * Let `Column::with_metadata` take any iterator of key-value pairs It took a `BTreeMap<String, String>`, so every caller had to build one: arrow hands out a `HashMap`, and a literal needed `.to_owned()` on each key and value. Taking `impl IntoIterator<Item = (impl Into<String>, impl Into<String>)>` lets all of them through, including `[("unit", "ns")]`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let `DynColumn::try_new` take a `Field` by value It took a `FieldRef`, so callers that had just built a `Field` had to wrap it in an `Arc` themselves. `impl Into<FieldRef>` accepts either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…c` (#54) * Wrap the name and metadata of `Column` in `Arc` Makes cloning a `Column` cheap: the name and the metadata are now shared, and `metadata_mut` copies on write. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix a redundant explicit doc link `BTreeMap` is now imported, so the explicit path makes rustdoc fail under `-D warnings`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

) `as_slice()` yields `&[ChunkId]` while `column[i]` yielded `&[u8; 16]`, so the same column reported two element types depending on how it was read — and `Deref` put `first()` (the newtype, through `as_slice`) right next to `get()` (the representation). The `primitive` arm already requires `Pod` and layout compatibility, so `Ref = Self` is sound the same way `values()` is: `must_cast_ref` on the representation's `value_ref`, size and alignment checked at compile time. `value()` / `get()` still hand out the representation's view — that is `LogicalType::Value`, load-bearing for the fallible arm, and a separate question. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add a benchmark for `TypedArray::slice` Measures the cost of one `slice` call per encoding, including the case that motivates the next commit: non-nullable list items whose child array still carries a validity buffer, which `downcast` has to scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Re-slice the downcast view instead of re-validating it `TypedArray::slice` sliced the arrow array and then re-ran `L::downcast` on the result: for a nested type that recurses into the children, and where a child carries a validity buffer it rescans that bitmap — work the whole array had already been validated against, and which slicing cannot invalidate (a slice reaches a subset of the rows). `LogicalType::slice_typed` is the new hook: it returns the view of the sliced rows, or `None` to fall back to `downcast`. The default is `None`, so a hand-written `LogicalType` impl keeps compiling and keeps today's behavior. Every encoding in the crate overrides it — the leaves slice themselves; the offset-based composites (list, list-view, map, dictionary, run) move the parent's window and share the child view; fixed-size lists move the child window too, since arrow re-bases their values on slice. slice of List<i64>, item null buffer 1.82µs -> 42ns slice of List<List<i64>> (100k rows) 85ns -> 60ns slice of i64 (1M) 34ns -> 34ns Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make `ErrorKind` the `source()` of `Error` `kind` had no `#[source]`, so the error chain stopped dead at `Error`: the cause that `ErrorKind::Conversion` stores as a `#[source] Box<dyn Error>` was unreachable programmatically, only printable. The doc on `From<Error> for ArrowError` already claimed the source chain was preserved. `thiserror` allows a field to be both interpolated and the source, so the message is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Name data types in errors with arrow's `Display`, not its `Debug` `Debug` on a nested `DataType` prints the whole inner `Field`: expected List(Field { data_type: Int64, nullable: true }) while `Display` prints `List(Int64)`. Every error that names a data type now uses `Display` — the `expected` strings the logical types produce, `DynColumn::try_new`, the derive, and the `actual` in `ErrorKind`'s and `ColumnError`'s own messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Attribute `extra_columns` errors to the struct being parsed `DynColumn::try_new` labels its errors `DynColumn`, so an error from the generated `extra_columns` collection would have named that instead of the `#[derive(Quiver)]` struct, unlike every other error the derive produces. The path is unreachable in practice — the field and array come out of a `RecordBatch`, which arrow has already checked the same two ways — so this is attribution only, and the validation stays as insurance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the `LogicalType` bound from `ColumnDesc`'s definition A descriptor holds no `L`, only a `PhantomData<fn() -> L>`, so the bound belongs on the impls that use it — as the Rust API guidelines put it. A bound on the data structure propagates outward instead: every user who writes `struct MySchema<L> { desc: ColumnDesc<L> }` has to repeat it, and the hand-written `Clone`/`Copy`/`Debug`/`PartialEq`/`Eq` had to carry it even though none of those traits needs anything of `L`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Write down the rule that keeps `Column`'s `Deref` safe An inherent method silently wins over a dereferenced one, so adding an inherent `Column::first()` some day would change the type and the meaning of every existing `column.first()` call site without a word from the compiler. State the rule next to the impl, and let `deref_to_slice` pin the slice-flavored types of the names that currently reach the slice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make `new_null`'s run-end panic length-independent, and fix the `Run` docs The panic was documented for run-end encoding but only fired for a non-empty column: `new_null_array` gives a zero-length `RunEndEncoded` empty run-ends and empty values, so there were no child nulls for `downcast` to reject. Check the data type up front instead. The docs also claimed `Option<Run<K, V>>` was unbuildable by any route. It is reachable through `optional()`, which is generic over every logical type — and reads every row as `Some`, there being no validity buffer for the `Option` layer to consult. Say so where a reader will look, and pin both behaviors with tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Pin `arrow_field_ref` to `arrow_field`, and say which is which The two methods had the same one-line doc summary, so rustdoc gave no hint which to reach for, and the `FieldRef` form had no test at all — the equivalence every caller assumes will want pinning when the implementation starts memoizing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix the newtype docs on when `primitive` is available `Pod` is about the layout, not the domain. The docs said a validating newtype "rarely accepts every bit pattern, so `primitive` is rarely available", which sends anyone with a plain newtype over `i64` and a `TryFrom` away from a bulk read that is both sound and correct: every bit pattern is a valid `Even` struct, and evenness is checked once per element at construction. What actually rules `Pod` out is a niche, as in `NonZero*` and `char`. The `Even` doctest now uses `primitive` and asserts `as_slice()`, so the claim is pinned rather than asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let every logical type out through `Column::into_dyn` `into_dyn` was bounded on `ConcreteType`, from when it built the field from `L::data_type()`. It now takes the data type from the array, so the bound is no longer earning anything — and it was shutting out exactly the users who need the dynamic exit: a `Column<AnyUtf8>` parsed from an unknown encoding had to hand-build the `Field`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Convert a `ColumnDesc` to a `DynColumnDesc` by value `ColumnDesc` is `Copy` and `to_dyn` takes `self`, but `DynColumnDesc::from` only accepted a reference, so the natural spelling did not compile. Also pin the documented metadata drop, which had no test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Give the owning array iterator the same combinator overrides `TypedArrayIter` overrides `count`, `last`, and `fold` to skip the `next`-based `Option` plumbing; `TypedArrayIntoIter` only had `size_hint` and `nth`, so `into_iter_owned().sum()` walked one `Option` at a time. The doc comment explaining the overrides sat above the type that had them, next to the one that did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Compare a `ColumnDesc`'s metadata as a map, not as a slice The declared metadata is semantically the map that lands on the arrow field, but it was compared as the ordered slice it is written as. So two descriptors that produce byte-identical `arrow_field`s could compare unequal (keys in a different order), and two that produce the same field could compare unequal the other way (a repeated key, which the field's map resolves last-wins). Order-insensitive and last-wins now, with no allocation in `eq`: the declared metadata is a handful of `&'static str` pairs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Shorten the `Deref` rule doc Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Rename the test constants that tripped the spell check `BA` reads as a typo for `BY`/`BE`, and the CI typos job agrees. The new names say what each descriptor differs in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…version (#58) * Add the `Transparent<T, Repr>` adapter: a domain-type tag with no conversion `Transparent<T, Repr>` behaves exactly like `Repr` — the same arrow data type, the same borrowed values, and the same owned values — with `T` only a type-level decoration. Unlike `As<T, Repr>` (which needs an infallible `T: From<Repr::Owned>` and converts on every owned read) and `try_newtype_data_type!` (which checks every value at column construction), `Transparent` validates nothing and converts nothing. That makes it the way to tag a column whose domain type is only `TryFrom` its representation, without paying for up-front validation; the caller converts the values it actually cares about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test that a `Transparent` column accepts a value its tag would reject `NonEmpty` rejects the empty string, but a `Transparent<NonEmpty, Utf8>` column runs no validation, so it takes one anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Move `As` and `Transparent` into their own files `newtype.rs` keeps the two macros and the standard-library types wired up with them. Doc links that relied on the adapters' imports now spell out their `crate::` paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add the 0.6.0 changelog Generated with `./scripts/generate_changelog.py --version 0.6.0`, then sorted into sections per `RELEASES.md`. Two hand edits the generator cannot make: * #24 is added by hand — it was merged into #23's branch before #23 landed, so it never appears in this branch's history. * #48 is listed under the name the release ships (`data_type()`), not the `datatype()` its PR title used before #50 renamed it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * nicer changelog --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

timsaucer

Looks great. It was a lot to review, and for the tests I did rely on the "trust me bro" that you recommended.

timsaucer · 2026-08-28T13:30:22Z

+[`DynColumn`](https://docs.rs/quiver/latest/quiver/struct.DynColumn.html) and
+[`DynColumnDesc`](https://docs.rs/quiver/latest/quiver/struct.DynColumnDesc.html).
+
 ## Supported `arrow` versions


Just below this do we need to update the readme version list to include 0.6?

timsaucer · 2026-08-28T13:42:33Z

+    /// field is not nullable but the array holds nulls.
+    pub fn try_new(field: impl Into<FieldRef>, array: ArrayRef) -> Result<Self, Error> {
+        let field = field.into();
+        let column = field.name().clone();


Allocation is done unconditionally but only used in errors, so we could potentially move this into the Error::new() calls.

timsaucer · 2026-08-28T13:47:26Z

+# NOTE: `bytemuck_derive` is pinned to 1.11 in `Cargo.lock`: 1.12 moved to `syn` 3,
+# which would duplicate the `syn` 2 that `quiver_derive` uses (see `cargo deny check bans`).
+bytemuck = { version = "1.16", features = ["derive", "must_cast"] }


We could potentially add bytemuck_derive to be explicitly a dependency here with "=1.11" to avoid a cargo update causing problems.

timsaucer · 2026-08-28T13:51:32Z

+    pub fn iter_owned(&self) -> impl Iterator<Item = L::Owned> + '_ {
+        self.iter().map(L::to_owned_value)
+    }


Did you want this one to be [must_use] also?

timsaucer · 2026-08-28T13:52:19Z

+    #[doc(alias = "non_nullable")]
+    pub fn try_required(self) -> Result<TypedArray<L::Required>, ColumnError> {


[must_use] also?

timsaucer · 2026-08-28T13:53:41Z

+    /// # Errors
+    /// Errors if the encoding fails, e.g. too many distinct values
+    /// for the dictionary key type.
+    pub fn try_from_nullable_values(


I'm going to stop commenting on each one because I'm sure you can have the robot adjust all of these if you want, but it's not obvious to me which you want to have as must_use and which not.

timsaucer · 2026-08-28T13:55:58Z

+    /// the arrow APIs that take a [`FieldRef`](arrow::datatypes::FieldRef).
+    #[must_use]
+    pub fn arrow_field_ref(&self) -> arrow::datatypes::FieldRef {
+        // TODO(emilk): it would be nice if this just `Arc::clone`d an existing `FieldRef` instead of allocating a new one on each call.


Might be worth a quick issue in the repo for this so that the work doesn't get lost over time

emilk and others added 11 commits August 27, 2026 17:08

Bump the version to 0.6.0

506a2b1

Opens the 0.6 release branch. `CHANGELOG.md` is generated from merged PR titles at release time, so it is not touched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Add ColumnDesc::arrow_field_ref (#38)

f793c49

emilk and others added 5 commits August 28, 2026 08:57

Add ColumnDesc::datatype() (#48)

a8c24f6

`Column` and `TypedArray` already report their arrow datatype statically; a descriptor could only do so via `arrow_field()`. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

This was referenced Aug 28, 2026

Add ColumnDesc::datatype() #48

Merged

Move DynColumn to its own file, with private, validated fields #49

Merged

Rename Datatype/datatype to DataType/data_type #50

Merged

This was referenced Aug 28, 2026

Add name() to the column descriptors and DynColumn #51

Merged

Give Column a name #52

Merged

emilk and others added 2 commits August 28, 2026 10:48

emilk mentioned this pull request Aug 28, 2026

Follow-ups from the 0.6 post-merge reviews #55

Merged

emilk and others added 4 commits August 28, 2026 12:59

emilk mentioned this pull request Aug 28, 2026

Add the 0.6.0 changelog #59

Merged

emilk marked this pull request as ready for review August 28, 2026 11:59

timsaucer self-requested a review August 28, 2026 12:50

timsaucer approved these changes Aug 28, 2026

View reviewed changes

Navigation Menu

Uh oh!

Release 0.6.0 - #30

Release 0.6.0#30
emilk wants to merge 23 commits into
mainfrom
release-0.6

emilk commented Aug 27, 2026 •

edited

Loading

Uh oh!

timsaucer left a comment

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

timsaucer Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		#[doc(alias = "non_nullable")]
		pub fn try_required(self) -> Result<TypedArray<L::Required>, ColumnError> {

Uh oh!

Conversation

emilk commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

What

Compatibility

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emilk commented Aug 27, 2026 •

edited

Loading