feat(zarr-metadata)!: a chunk grid is judged against the shape it chunks (layer 4b)

packages/zarr-metadata/README.md

@@ -65,8 +65,9 @@ configuration its definition refuses is refused, and a key it does not
 declare is reported as `unknown_key`. A name nothing in the scope claims is
 left unjudged, and whether to support it is the consumer's decision. A v3
 fill value is judged against the data type it names, by that data type's
+definition, and the chunk grid against the shape, by the grid's
 definition. The validators do not judge a codec against the array it is
-handed, or a chunk grid against the shape.
+handed.
 
 The Pydantic integration's generated JSON Schemas express independently
 checkable document structure and field constraints, but they are not a

packages/zarr-metadata/changes/367.bugfix.md

@@ -0,0 +1,3 @@
+`ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1
+for a dimension of length 0, where it wrote 0: the regular grid asks for
+chunk lengths greater than zero, and `zarr` does not open a grid with one.

packages/zarr-metadata/changes/367.feature.md

@@ -0,0 +1,7 @@
+**Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by
+the grid's definition. A regular grid whose `chunk_shape` does not have
+one length per dimension of the shape, or has a length of 0 for a
+dimension that is not empty, and a rectilinear grid whose `chunk_shapes`
+does not have one entry per dimension, or whose chunk lengths fall short
+of their dimension, each have a problem in `chunk_grid.configuration`,
+where the package accepted them before.

packages/zarr-metadata/docs/index.md

@@ -80,8 +80,9 @@ configuration its definition refuses is refused, and a key it does not
 declare is reported as `unknown_key`. A name nothing in the scope claims is
 left unjudged, and whether to support it is the consumer's decision. A v3
 fill value is judged against the data type it names, by that data type's
+definition, and the chunk grid against the shape, by the grid's
 definition. The validators do not judge a codec against the array it is
-handed, or a chunk grid against the shape.
+handed.
 
 ## Scope
 

packages/zarr-metadata/src/zarr_metadata/model/__init__.py

@@ -5,7 +5,8 @@
 structure and, in a v3 document, read each extension point (codecs, chunk
 grids, data types, ...) through the definition that claims its name in a
 scope, `CORE_AND_EXTENSIONS` unless a `context` is passed, and judge the
-fill value against the data type it names. Each document concept gets a
+fill value against the data type it names and the chunk grid against
+the shape. Each document concept gets a
 `validate_*` function returning every problem found (a tuple of
 `ValidationProblem`, each with a machine-readable `kind`), an `is_*` type
 guard, and a `parse_*` function that narrows or raises

packages/zarr-metadata/src/zarr_metadata/model/_array.py

@@ -191,22 +191,26 @@ def create_default(cls, **overrides: Unpack[ZarrV3ArrayMetadataPartial]) -> Zarr
         analog of `list()` returning `[]`. Any field can be overridden by keyword
         (the same fields accepted by `update`). Overriding `shape` without
         `chunk_grid` derives a consistent default grid: one regular chunk
-        covering the array (`chunk_shape` equal to `shape`).
+        covering the array (`chunk_shape` equal to `shape`, with a length of
+        1 for a dimension of length 0, since a chunk length is at least 1:
+        https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-grids/regular-grid/index.rst#L40).
 
         The derivation is deliberately one-way. A user-supplied `chunk_grid`
         is an extension point and is taken verbatim — deriving `shape` from
         it would require interpreting the grid's configuration, which this
         layer never does (and cannot do for unrecognized grid names). So
         overriding `chunk_grid` without `shape` keeps the scalar default
-        `shape=()`, and consistency between the two is the caller's
-        responsibility. So is a fill value for an overridden `data_type`:
+        `shape=()`, which a grid of another rank does not fit: consistency
+        between the two is the caller's responsibility, so pass them
+        together. So is a fill value for an overridden `data_type`:
         the default `fill_value` is `0`, which a data type whose fill value
         is not an integer -- `bool`, `string`, a complex or struct type --
         refuses, so pass the two together.
         """
         if "shape" in overrides and "chunk_grid" not in overrides:
+            chunk_shape = tuple(max(length, 1) for length in overrides["shape"])
             overrides["chunk_grid"] = ZarrV3NamedConfig(
-                name="regular", configuration={"chunk_shape": tuple(overrides["shape"])}
+                name="regular", configuration={"chunk_shape": chunk_shape}
             )
         default = cls(
             shape=(),

packages/zarr-metadata/src/zarr_metadata/model/_validation.py

@@ -5,8 +5,8 @@
 each extension point through the definition that claims its name in a
 scope, so a configuration its definition refuses is refused here too. A
 name nothing in the scope claims is left unjudged. A v3 fill value is
-judged against the data type it names; a codec against the array it is
-handed, and a grid against the shape, are not judged here. Each concept
+judged against the data type it names, and the chunk grid against the
+shape; a codec against the array it is handed is not judged here. Each concept
 gets a `validate_*` function returning every problem found, an `is_*`
 type guard, and a `parse_*` function that narrows or raises
 `MetadataValidationError`. The guards are `TypeGuard`s,
@@ -45,6 +45,7 @@
     Definition,
     Resolved,
     StorageTransformerDefinition,
+    chunk_grid_problems,
     fill_value_problems,
     resolve,
 )
@@ -183,19 +184,22 @@ def _is_int_sequence(value: object) -> TypeGuard[Sequence[int]]:
     )
 
 
-def _validate_dim_sequence(doc: Mapping[object, object], key: str) -> tuple[ValidationProblem, ...]:
-    """Validate a dimension sequence (`shape` / `chunks`) if present in `doc`.
+def _dimension_lengths(
+    doc: Mapping[object, object], key: str
+) -> tuple[tuple[int, ...] | None, tuple[ValidationProblem, ...]]:
+    """The dimension lengths `doc` holds at `key` (`shape`, `chunks`), and every problem with them.
 
-    Dimension lengths are non-negative integers.
+    Dimension lengths are non-negative integers; the lengths are None when
+    `doc` holds none at `key`, or ones with a problem.
     """
     if key not in doc:
-        return ()
+        return None, ()
     value = doc[key]
     if not _is_int_sequence(value):
-        return (ValidationProblem((key,), "expected a sequence of int", "invalid_type"),)
+        return None, (ValidationProblem((key,), "expected a sequence of int", "invalid_type"),)
     if any(item < 0 for item in value):
-        return (ValidationProblem((key,), "expected non-negative integers", "invalid_value"),)
-    return ()
+        return None, (ValidationProblem((key,), "expected non-negative integers", "invalid_value"),)
+    return tuple(value), ()
 
 
 def _is_dtype_v2(value: object) -> bool:
@@ -349,9 +353,11 @@ def validate_array_metadata_v3(
     Its structure, and each extension point read through the definition
     that claims its name in `context`: a gzip `level` out of range, a key a
     codec's configuration does not declare. The fill value is judged
-    against the data type as `context` read it: an `int8` fill value of 300.
-    A name nothing in `context` claims is left unjudged, with any fill value
-    of it. Unknown top-level keys are allowed (they map
+    against the data type as `context` read it -- an `int8` fill value of
+    300 -- and the chunk grid against the shape: a regular grid with a
+    chunk length for each of two dimensions, over an array of three. A name
+    nothing in `context` claims is left unjudged, with any fill value of
+    it. Unknown top-level keys are allowed (they map
     to `extra_fields`); a reader must understand each one that does not
     say `must_understand: false`, which the model reports as
     `must_understand_fields`.
@@ -363,7 +369,8 @@ def validate_array_metadata_v3(
     problems.extend(_validate_other_members(doc, ARRAY_METADATA_STANDARD_KEYS_V3))
     problems.extend(_check_literal(doc, "zarr_format", 3))
     problems.extend(_check_literal(doc, "node_type", "array"))
-    problems.extend(_validate_dim_sequence(doc, "shape"))
+    shape, shape_problems = _dimension_lengths(doc, "shape")
+    problems.extend(shape_problems)
     # Each extension point is read by `resolve`, which judges its envelope
     # -- every extension *point* must be understood, so a `must_understand`
     # of `false` is refused at each: ignoring a codec gives wrong bytes as
@@ -389,6 +396,9 @@ def validate_array_metadata_v3(
             )
         else:
             problems.extend(_prefix("fill_value", validate_json(doc["fill_value"])))
+    # The chunk grid is judged against the shape, once both are read.
+    if "chunk_grid" in read and shape is not None:
+        problems.extend(chunk_grid_problems(read["chunk_grid"], shape, ("chunk_grid",)))
     for key, kind in _EXTENSION_LISTS_V3:
         if key in doc:
             entries = doc[key]
@@ -410,7 +420,6 @@ def validate_array_metadata_v3(
         # field-level loc, not per-bad-item locs; per-index locs are reserved for
         # the metadata-field lists (codecs, storage_transformers).
         names = doc["dimension_names"]
-        shape = doc.get("shape")
         if not _is_array(names):
             problems.append(
                 ValidationProblem(("dimension_names",), "expected a sequence", "invalid_type")
@@ -421,7 +430,7 @@ def validate_array_metadata_v3(
                     ("dimension_names",), "expected items of str or None", "invalid_type"
                 )
             )
-        elif _is_int_sequence(shape) and len(names) != len(shape):
+        elif shape is not None and len(names) != len(shape):
             problems.append(
                 ValidationProblem(
                     ("dimension_names",),
@@ -474,19 +483,11 @@ def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]:
     problems: list[ValidationProblem] = list(_missing_keys(ARRAY_METADATA_REQUIRED_KEYS_V2, doc))
     problems.extend(_validate_other_members(doc, ARRAY_METADATA_STANDARD_KEYS_V2))
     problems.extend(_check_literal(doc, "zarr_format", 2))
-    shape_problems = _validate_dim_sequence(doc, "shape")
-    chunks_problems = _validate_dim_sequence(doc, "chunks")
+    shape, shape_problems = _dimension_lengths(doc, "shape")
+    chunks, chunks_problems = _dimension_lengths(doc, "chunks")
     problems.extend(shape_problems)
     problems.extend(chunks_problems)
-    shape = doc.get("shape")
-    chunks = doc.get("chunks")
-    if (
-        len(shape_problems) == 0
-        and len(chunks_problems) == 0
-        and _is_int_sequence(shape)
-        and _is_int_sequence(chunks)
-        and len(shape) != len(chunks)
-    ):
+    if shape is not None and chunks is not None and len(shape) != len(chunks):
         problems.append(
             ValidationProblem(
                 ("chunks",),

packages/zarr-metadata/src/zarr_metadata/v3/_definition.py

@@ -83,8 +83,8 @@
 Problems: TypeAlias = tuple[ValidationProblem, ...]
 
 
-def no_rules(configuration: object) -> Iterator[ValidationProblem]:
-    """The rules of a definition with none: every well-typed configuration is allowed."""
+def no_rules(*_: object) -> Iterator[ValidationProblem]:
+    """The rules of a definition with none, of any kind: everything well typed is allowed."""
     yield from ()
 
 
@@ -93,13 +93,6 @@ def unchanged(configuration: T) -> T:
     return configuration
 
 
-def no_fill_value_rules(
-    configuration: object, nested: Nested, value: object
-) -> Iterator[ValidationProblem]:
-    """The fill value rules of a data type with none: every fill value of its JSON shape is allowed."""
-    yield from ()
-
-
 class EmptyConfiguration(TypedDict, closed=True):
     """The configuration of a definition with nothing to configure, written as its bare name."""
 
@@ -276,7 +269,7 @@ class DataTypeDefinition(Definition[C]):
 
     fill_value: object = JSONValue
     """The JSON shape of a fill value, as an annotation: `Int8FillValue`."""
-    fill_value_rules: Callable[[C, Nested, Any], Iterable[ValidationProblem]] = no_fill_value_rules
+    fill_value_rules: Callable[[C, Nested, Any], Iterable[ValidationProblem]] = no_rules
     """What the spec disallows in a fill value of that shape, located in it."""
 
     def _refusal(self) -> str | None:
@@ -300,7 +293,18 @@ def _fill_value_parser(annotation: object) -> Parser:
 
 @dataclass(frozen=True, kw_only=True, slots=True)
 class ChunkGridDefinition(Definition[C]):
-    """A chunk grid."""
+    """A chunk grid, and the arrays it fits.
+
+    `shape_rules` is what the spec disallows in a grid of this
+    configuration over an array of a given shape: a dimension with no
+    chunk length, chunks that fall short of one. It is handed the
+    configuration, the fields it holds as the scope read them, and the
+    shape, and locates its problems in the configuration. A grid that
+    says nothing of the shape fits every one.
+    """
+
+    shape_rules: Callable[[C, Nested, tuple[int, ...]], Iterable[ValidationProblem]] = no_rules
+    """What the spec disallows in this grid over an array of a shape, located in the configuration."""
 
 
 @dataclass(frozen=True, kw_only=True, slots=True)
@@ -703,6 +707,28 @@ def fill_value_problems(
     return (*problems, *refused)
 
 
+def chunk_grid_problems(
+    chunk_grid: Resolved[ChunkGridDefinition[Any]], shape: tuple[int, ...], loc: Loc = ()
+) -> Problems:
+    """What is wrong with `chunk_grid`, a chunk grid field a scope read, over an array of `shape`.
+
+    Judged by the grid's shape rules, which locate their problems in the
+    configuration, under `loc`, where the field sits: a regular grid with
+    a chunk length for each of two dimensions, over an array of three. A
+    grid the scope did not read, out of scope or invalid, is left
+    unjudged.
+    """
+    definition = chunk_grid.definition
+    configuration = chunk_grid.configuration
+    if definition is None or configuration is None:
+        return ()
+    return _ruled(
+        definition,
+        lambda: definition.shape_rules(configuration, chunk_grid.nested, shape),
+        (*loc, "configuration"),
+    )
+
+
 def resolve(
     data: object, kind: type[D], context: Context, loc: Loc = ()
 ) -> tuple[Resolved[D], Problems]:
@@ -912,11 +938,11 @@ def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
     "Unread",
     "as_kind",
     "canonicalize",
+    "chunk_grid_problems",
     "configuration_of",
     "fill_value_problems",
     "kind_of",
     "named_configuration",
-    "no_fill_value_rules",
     "no_rules",
     "resolve",
     "spelled",

packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py

@@ -11,7 +11,7 @@
 
 from zarr_metadata._json import ValidationProblem
 from zarr_metadata._typed_json import Loc
-from zarr_metadata.v3._definition import ChunkGridDefinition
+from zarr_metadata.v3._definition import ChunkGridDefinition, Nested
 
 RECTILINEAR_CHUNK_GRID_NAME: Final = "rectilinear"
 """The `name` field value of the rectilinear chunk grid."""
@@ -123,11 +123,43 @@ def _canonical(
     )
 
 
+def _shape_rules(
+    configuration: RectilinearChunkGridConfiguration, nested: Nested, shape: tuple[int, ...]
+) -> Iterator[ValidationProblem]:
+    """Chunk lengths for each of the array's dimensions, which cover it.
+
+    "The length of `chunk_shapes` MUST match the number of dimensions of
+    the array", and "The sum of the edge lengths MUST equal or exceed `L`"
+    (https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md?plain=1#L62-L91).
+    A bare integer repeats until it covers the dimension, so it always does.
+    """
+    chunk_shapes = configuration["chunk_shapes"]
+    if len(chunk_shapes) != len(shape):
+        yield ValidationProblem(
+            ("chunk_shapes",),
+            f"expected one chunk_shapes entry per dimension of shape, got {len(chunk_shapes)}",
+            "invalid_value",
+        )
+        return
+    for axis, (spec, extent) in enumerate(zip(chunk_shapes, shape, strict=True)):
+        if isinstance(spec, int):
+            continue
+        covered = sum(entry if isinstance(entry, int) else entry[0] * entry[1] for entry in spec)
+        if covered < extent:
+            yield ValidationProblem(
+                ("chunk_shapes", axis),
+                f"expected chunk lengths that cover the dimension's length {extent}, "
+                f"got lengths summing to {covered}",
+                "invalid_value",
+            )
+
+
 RECTILINEAR_CHUNK_GRID: Final = ChunkGridDefinition(
     name=RECTILINEAR_CHUNK_GRID_NAME,
     configuration=RectilinearChunkGridConfiguration,
     rules=_rules,
     canonical=_canonical,
+    shape_rules=_shape_rules,
 )
 """The `rectilinear` chunk grid."""