feat(zarr-metadata): the model reads each extension point through its definition (layer 3)

[d-v-b]

🤖 AI text below 🤖

Mirror of zarr-developers#4436, where this is reviewed: the same head commit, on main now that layers 0 to 2 have merged upstream. Review fixes land on the upstream branch and this branch follows. The description below is the upstream one; the earlier description, with the review history and the 40,000-document differential, is in this description's edit history.

The model reads each extension point of a v3 document through its definition. validate_array_metadata_v3 judged an array document's structure and left every configuration unread, so a gzip level of 99, or a key a codec does not declare, passed. It now reads the data type, the chunk grid, the chunk key encoding, each codec and each storage transformer with resolve (zarr-developers#4434), in a scope: CORE_AND_EXTENSIONS unless a context is passed. A name nothing in scope claims is left unjudged, which keeps the format open.

from zarr_metadata.model import validate_array_metadata_v3

doc["codecs"] = [{"name": "bytes"}, {"name": "gzip", "configuration": {"level": 99}}]
validate_array_metadata_v3(doc)
# (ValidationProblem(loc=('codecs', 1, 'configuration', 'level'), ..., kind='invalid_value'),)

What reads in a scope. The is_* and parse_* beside it read the same way, as do the v3 group validators and each array an inline consolidated_metadata holds, with problems located down to the member: ('consolidated_metadata', 'metadata', 'a', 'codecs', 1, 'configuration', 'level'). Each takes context=, and so do the v3 model classes' from_json, from_key_value and to_key_value. to_key_value validates what it writes in that scope, so the model never writes what its reader in the same scope refuses, and a reader that substitutes its own reading of a codec can build the model from a document its scope accepts and write it back. The pydantic types read in CORE_AND_EXTENSIONS, since a field type holds no scope; the module docstring says so, and how a reader with its own scope calls from_json instead.

What breaks. A document whose configurations its definitions refuse, which main accepts, now has problems: is_* says no, and parse_*, from_json and from_key_value raise. The fragment is marked Breaking:. A key a configuration does not declare is reported as unknown_key and the field is still read, so a consumer that tolerates one can filter problems by kind.

Docs. The README and the docs index said the model validators do not interpret extension names or configurations. They now say what the validators read, what they leave unjudged, and that no rule reads one field against another. The definition guide shows a scope reading a whole document.

Not checked here. Every rule that reads one field against another waits for the next layer:

• a fill value against the data type;
• a bytes codec's endian for a multi-byte type;
• a transpose order against the rank;
• a pipeline's order and its one array-to-bytes codec;
• sharding's inner pipelines and divisibility;
• a grid's rank against the shape;
• a zero chunk extent on a non-empty dimension;
• scale_offset's scale and offset against the data type;
• v2's chunks against its shape.

Cost. Validating a document now reads its configurations. On CPython 3.13, best of several runs, a simple array document takes about 20 µs where main takes about 5 µs, and a sharded one about 45 µs where main takes about 12 µs. A group's from_json validates each array its consolidated metadata holds three times, five inside a nested group, as main does; each pass now costs this much more. Validating each array once is a follow-up.

Not here.

• The JSON schema zarr_metadata.pydantic generates describes each extension point's envelope, not its configuration, so it accepts a gzip level of 99 that the runtime refuses.
• The model reports a member its document does not declare as invalid_value, where the checker reports unknown_key, even for the inline consolidated envelope, whose TypedDict is closed. Making those kinds agree changes the kinds main reports, not its verdicts, so it is its own change. unknown_key is on main since feat(zarr-metadata): check, JSON against a TypedDict as the typing spec reads it zarr-developers/zarr-python#4432, so that change should land before the next release.

Also here. The must_understand fragment on main, 4433.feature.md, is renamed 4434.feature.1.md: that change merged with zarr-developers#4434, and zarr-developers#4433 now has nothing left to merge.

Review question: is reading each extension point through its definition, in a scope the caller can replace, the right shape for the document's reader?

The stack. Layers 0 (zarr-developers#4420, zarr-developers#4421, zarr-developers#4422), 1 (zarr-developers#4432) and 2 (zarr-developers#4434, which carried zarr-developers#4433's must_understand change) have merged. This is layer 3, based on main. Layer 4, the field among fields, reads the rules listed above and is not written yet.

🤖 Generated with Claude Code

[d-v-b]

🤖 AI text below 🤖

Layer 3 merged upstream as zarr-developers#4436, so this fork copy is closed. Layer 4 (#366 → #367 → #368 → #369) goes upstream next.