feat(zarr-metadata): incremental v3 array builder, extension-point table, TypeIs codec guards

packages/zarr-metadata/changes/319.feature.1.md

@@ -0,0 +1,18 @@
+Added `zarr_metadata.v3._extension_points`: a table of the v3 extension
+points — `data_type`, `chunk_grid`, `chunk_key_encoding`, `codecs`,
+`storage_transformers` — recording, per identifier, where it was
+standardized and where its definition lives, and per point, whether
+`must_understand: false` is permitted there and whether the field holds
+one entity or a sequence.
+
+Keyed by `(field, name)`, because names are unique only *within* an
+extension point: `bytes` is both a core codec and a registered extension
+data type.
+
+Provenance is `CORE`, `REGISTERED`, or `PROPOSED`; `zstd` is marked
+proposed because its cited specification is still an open pull request.
+
+`canonical_name(field, name)` is identity except for raw-byte data types:
+all `r<N>` spellings, including invalid ones such as `r12`, map to
+`RAW_BYTES_FAMILY`. This lets validation report malformed family members
+instead of treating them as unknown extensions.

packages/zarr-metadata/changes/319.feature.2.md

@@ -0,0 +1,8 @@
+Added `zarr_metadata.v3.codec.kind`: codec kind classification. One
+branded union per spec pipeline kind over the concrete codec types
+(`ArrayArrayCodecMetadata`, `ArrayBytesCodecMetadata`,
+`BytesBytesCodecMetadata`, plus `KnownCodecMetadata` and the paired
+`*_CODEC_NAMES` constants). Shape-exact `TypeIs` guards narrow canonical
+codec metadata to those unions. `codec_kind_of_name` classifies known
+names without validating object shape and returns `None` for unknown
+names. All names are re-exported from `zarr_metadata.v3.codec`.

packages/zarr-metadata/changes/319.feature.4.md

@@ -0,0 +1,16 @@
+Added `zarr_metadata.builder`: incremental, validated construction of v3
+array metadata documents over the plain JSON TypedDict shapes.
+`ZarrV3ArrayMetadataBuilder` accumulates a
+`ZarrV3ArrayMetadataJSONPartial` and returns updated copies.
+`with_fields(**kwargs)` types standard fields; PEP 728 checkers also accept
+extension fields, while `with_extension` works across checkers.
+`without_fields` removes keys, properties return `T | UNSET`, `build()` returns
+a complete validated document, and `to_partial_json()` returns the
+current fragment. Inputs are copied and JSON arrays normalize to tuples.
+
+Composition rules fire after each change once their dependencies are
+present. They check fill values, known entity shapes, codec pipelines,
+dimension counts, chunk grids, transpose codecs, and sharding. Unknown
+entity names are left unjudged; known names must use their canonical
+shape. A failing update raises one `MetadataValidationError` containing
+all problems found.

packages/zarr-metadata/docs/api/builder.md

@@ -3,3 +3,8 @@ title: builder
 ---
 
 ::: zarr_metadata.builder
+
+See `examples/build_v3_array.py` in the package for incremental
+construction, rejected invalid updates, JSON round-tripping, and
+consolidated metadata. It ships in the sdist and is executed by the test
+suite, so it cannot drift from the API it demonstrates.

packages/zarr-metadata/docs/api/index.md

@@ -13,7 +13,8 @@ The package is organized to mirror the structure of the Zarr specifications:
   ordering, chunk geometry), plus whole-document `validate`/`is`/`parse`
   trios combining structure and composition
 - [`zarr_metadata.builder`](builder.md) — validated construction:
-  one-shot `create_*` factories, one per document type
+  one-shot `create_*` factories per document type and an incremental
+  builder for v3 arrays
 - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
   over the models
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents

packages/zarr-metadata/examples/build_v3_array.py

@@ -0,0 +1,92 @@
+"""Construct, serialize, and validate a Zarr v3 metadata hierarchy.
+
+Run from ``packages/zarr-metadata`` with::
+
+    uv run python examples/build_v3_array.py
+
+The example uses only public APIs. Assertions make it useful as an executable
+smoke test as well as a starting point for applications.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING
+
+from zarr_metadata.builder import (
+    ZarrV3ArrayMetadataBuilder,
+    create_zarr_v3_consolidated_metadata_json,
+    create_zarr_v3_group_metadata_json,
+)
+from zarr_metadata.model import MetadataValidationError
+from zarr_metadata.rules import Valid, check_group_metadata_v3
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+
+def expect_rejected(label: str, operation: Callable[[], object]) -> None:
+    """Run one deliberately invalid operation and show why it was rejected."""
+    try:
+        operation()
+    except MetadataValidationError as error:
+        print(f"Rejected {label}: {error.problems[0]}")
+    else:  # pragma: no cover - this script is also an executable assertion
+        raise AssertionError(f"Expected {label} to be rejected")
+
+
+def main() -> None:
+    # Composition errors are rejected as soon as all fields involved are known.
+    expect_rejected(
+        "an out-of-range uint8 fill value",
+        lambda: ZarrV3ArrayMetadataBuilder().with_fields(data_type="uint8", fill_value=300),
+    )
+
+    # Known extension names are checked against their canonical configuration.
+    expect_rejected(
+        "an invalid chunk-key separator",
+        lambda: ZarrV3ArrayMetadataBuilder().with_fields(
+            chunk_key_encoding={"name": "default", "configuration": {"separator": "!"}}
+        ),
+    )
+
+    array = (
+        ZarrV3ArrayMetadataBuilder()
+        .with_fields(zarr_format=3, node_type="array", shape=(100, 200))
+        .with_fields(data_type="uint16", fill_value=0)
+        .with_fields(
+            chunk_grid={"name": "regular", "configuration": {"chunk_shape": (10, 20)}},
+            chunk_key_encoding={"name": "default", "configuration": {"separator": "/"}},
+            codecs=(
+                {"name": "transpose", "configuration": {"order": (1, 0)}},
+                {"name": "bytes", "configuration": {"endian": "little"}},
+                "crc32c",
+            ),
+            dimension_names=("y", "x"),
+            attributes={"units": "counts"},
+        )
+        .build()
+    )
+
+    # JSON round-tripping changes tuples into lists; the read-side API normalizes
+    # them back before returning a typed, composition-valid document.
+    loaded_array = json.loads(json.dumps(array))
+    consolidated = create_zarr_v3_consolidated_metadata_json(
+        kind="inline", must_understand=False, metadata={"measurements": loaded_array}
+    )
+    group = create_zarr_v3_group_metadata_json(
+        zarr_format=3,
+        node_type="group",
+        attributes={"title": "Example hierarchy"},
+        extensions={"consolidated_metadata": consolidated},
+    )
+
+    checked = check_group_metadata_v3(json.loads(json.dumps(group)))
+    assert isinstance(checked, Valid)
+    assert checked.document == group
+
+    print(json.dumps(checked.document, indent=2))
+
+
+if __name__ == "__main__":
+    main()

packages/zarr-metadata/justfile

@@ -21,7 +21,7 @@ pyright_version := "1.1.404"
 # stdlib it cannot parse, so pin the interpreter to match CI.
 # Type-check the package sources
 typecheck:
-    uv run --python 3.11 --group test --with 'pyright=={{ pyright_version }}' pyright src
+    uv run --python 3.11 --group test --with 'pyright=={{ pyright_version }}' pyright src examples
 
 # Run everything CI runs for this package
 check: lint typecheck test docs-check

packages/zarr-metadata/pyproject.toml

@@ -74,8 +74,9 @@ packages = ["src/zarr_metadata"]
 # An allowlist, so nothing that merely happens to sit in the package directory
 # — a scratch script, a stray notebook — can ride along in a release. The list
 # keeps an sdist self-testing and self-documenting: every fixture this suite
-# reads is a JSON file sitting next to the test module that loads it, so
-# `/tests` is the whole test dependency, and `/docs` plus `/mkdocs.yml` are a
+# reads is a JSON file sitting next to the test module that loads it; the
+# one exception is `/examples`, which `tests/test_examples.py` executes, so
+# both ship, and `/docs` plus `/mkdocs.yml` are a
 # self-contained site (mkdocstrings reads `src`, nothing reaches outside the
 # package) so `just docs-check` runs from an unpacked sdist too. `changes/`
 # and `.readthedocs.yaml` are deliberately absent: towncrier fragments are
@@ -85,6 +86,7 @@ packages = ["src/zarr_metadata"]
 include = [
     "/src",
     "/tests",
+    "/examples",
     "/docs",
     "/mkdocs.yml",
     "/justfile",
@@ -129,7 +131,7 @@ checks = [
 # class attributes (microsoft/pyright#11115), which zarr_metadata.model._sentinel
 # relies on. Use the same pin locally; unpin when the fix lands.
 [tool.pyright]
-include = ["src"]
+include = ["src", "examples"]
 enableExperimentalFeatures = true
 typeCheckingMode = "strict"
 pythonVersion = "3.11"

packages/zarr-metadata/src/zarr_metadata/builder/__init__.py

@@ -5,10 +5,14 @@
 at literal-keyword call sites, and the runtime pass normalizes the input
 and applies structural and composition validation, raising one
 `MetadataValidationError` carrying every problem.
+`ZarrV3ArrayMetadataBuilder` supports incremental v3 array construction,
+running applicable composition rules after each update and checking
+completeness at `build()`.
 
 Use `zarr_metadata.rules` to validate documents read from storage.
 """
 
+from zarr_metadata.builder._array_v3 import ZarrV3ArrayMetadataBuilder
 from zarr_metadata.builder._create import (
     create_zarr_v2_array_metadata_json,
     create_zarr_v2_consolidated_metadata_json,
@@ -21,6 +25,7 @@
 )
 
 __all__ = [
+    "ZarrV3ArrayMetadataBuilder",
     "create_zarr_v2_array_metadata_json",
     "create_zarr_v2_consolidated_metadata_json",
     "create_zarr_v2_group_metadata_json",