feat(zarr-metadata)!: a chunk grid is judged against the shape it chunks (layer 4b)

[d-v-b]

🤖 AI text below 🤖

Upstream: zarr-developers#4441, a draft on main now that layer 3 (zarr-developers#4436) has merged. This fork PR stays as the stacked review copy.

Layer 4b of the stack, on top of #366 (4a, fill values): a v3 array's chunk_grid is judged against its shape, by the grid's definition. It is the second of the four pieces of layer 4 (4a fill values, 4b grid against shape, 4c the codec pipeline, 4d sharding).

document["shape"] = [4, 4]
document["chunk_grid"] = {"name": "regular", "configuration": {"chunk_shape": [4]}}
validate_array_metadata_v3(document)
# (ValidationProblem(loc=('chunk_grid', 'configuration', 'chunk_shape'), message='expected one chunk length per dimension of shape, got 1', kind='invalid_value'),)

What a grid knows. ChunkGridDefinition gains shape_rules(configuration, nested, shape): what the spec disallows in a grid of that configuration over an array of a given shape, located in the configuration. It takes the same arguments as 4a's fill_value_rules: the configuration, the fields it holds as the scope read them, and the thing judged. A grid that says nothing of the shape fits every one, so a third-party grid is unaffected. The two package grids:

• regular: one chunk length per dimension (regular grid), and 0 only for a dimension of length 0 (core: "non-zero when the corresponding dimensions of the arrays have non-zero length"). A chunk longer than its dimension is fine.
• rectilinear: one chunk_shapes entry per dimension, with chunk lengths that sum to at least the dimension's length (extension). A bare integer repeats until it covers, so it always does. A run-length entry counts as length times count and is never expanded. [] covers a dimension of length 0.

Where it is judged. validate_array_metadata_v3 judges the grid once both the grid and the shape are read. A grid the scope did not read, being out of scope or invalid, is left unjudged. So is a grid beside a shape that fails its own check, so no problem is reported twice. The check reaches from_json, from_key_value, to_key_value, the pydantic types and each array in an inline consolidated_metadata. Reading the shape now has one owner, which returns the lengths or None with its problems. The v3 grid and dimension_names checks use it, and so does the v2 rank check between shape and chunks.

chunk_grid_problems stays private, in v3._definition. 4c's door returns each axis's chunk lengths along with the problems, and replaces it.

Breaking. A document whose grid does not fit its shape now has a problem in chunk_grid.configuration, where the package accepted it before. create_default(chunk_grid=...) without shape keeps the scalar default shape=(), which a grid of another rank does not fit, so that model now fails at to_key_value. The docstring says to pass the two together.

A fix on the way. create_default(shape=...) set chunk_shape equal to shape, so an empty dimension got a chunk length of 0. zarr refuses to open that grid: "integer chunk edge length must be >= 1". The default grid now has a chunk length of 1 there, which still fits the shape. This is its own commit, with a bugfix fragment.

Choices worth a look

• A regular chunk length of 0 on a dimension of length 0 is accepted when read. The core spec's condition allows it, and zarr-python 3.0 and 3.1 wrote it, although the regular grid page says chunk sizes must be greater than zero. The package's own writer no longer produces it (above).
• zarr-python refuses a rectilinear [] for a dimension of length 0 ("has no chunk edges"). This PR accepts it, since there is nothing to cover.
• The v2 create_default has the same derivation: chunks equals shape, with 0 for an empty dimension. zarr opens that document, so this PR leaves it alone.
• shape_rules returns only problems. In 4c the lengths come from a separate function, which is only called on a grid its shape_rules accepted. The design review considered a single chunks -> (lengths, problems) hook and kept the split, for the same reason rules and canonical are split: the rules report problems, and derivation works from what they accepted.

Reviews. Two, with separate lenses. The correctness review found no bug in the rules. A Hypothesis differential against a reference written from the spec text agreed on 8,000 examples, and its reach was measured: rank mismatches, 0 on an empty dimension, uncovered dimensions, run-length entries, exact cover and [] for length 0 all came up. Canonical and written rectilinear spellings agreed on 3,000 examples. zarr-python agrees everywhere except [], noted above. The review's one finding was the chunk length of 0 from create_default. The design review's changes are in too:

• chunk_grid_problems stays private.
• One no_rules(*_) default serves every hook, instead of one per arity.
• Reading the shape has one owner.
• Messages and citations are fixed.
• The tests are split one per case.

The stack: 0 and 1 merged upstream (zarr-developers#4420, zarr-developers#4421, zarr-developers#4422, zarr-developers#4432), 2 merged with 1b as zarr-developers#4434, 3 is #355 (upstream zarr-developers#4436), 4a is #366, 4b is this PR, then 4c the codec pipeline and 4d sharding.

🤖 Generated with Claude Code

[d-v-b]

🤖 AI text below 🤖

Merged upstream: all of layer 4 landed as zarr-developers#4443. This fork review copy is closed.