fix: make the chunk normalizer the one judge of a chunk specification by d-v-b · Pull Request #4376 · zarr-developers/zarr-python

d-v-b · 2026-09-18T16:54:17Z

🤖 AI text below 🤖

Follow-up to #4374. Target: 3.4.1 (patch). Independent of #4334, #4375 and #4377.

Problem

The chunk normalizer (normalize_chunks_nd → normalize_chunks_1d) was not the only judge of a chunks=/shards= specification. A separate duck-typed classifier, _is_rectilinear_chunks, ran on the raw input at three sites (AsyncArray._create, init_array, resolve_outer_and_inner_chunks) to decide whether the specification was rectilinear, so that the Zarr format 2 and sharding restrictions could fire. Two opinions on the same input is the shape of the bug in #4374, and they could disagree: a 0-d NumPy array counted as rectilinear because it has __iter__, and ChunkGrid.from_sizes collapsed a uniform edge list to a regular dimension while normalize_chunks_1d kept it rectilinear.

Changes

One judge. _is_rectilinear_chunks is deleted. Each site normalizes first and asks the resulting ChunkGrid (is_regular). Regular and rectilinear shard specifications go through the same normalizer.
One integer rule in the normalizer (_chunk_int): a chunk size is anything Python's integer protocol (operator.index) accepts: int, NumPy integer scalars and 0-d integer arrays. Floats, arrays with dimensions and NumPy booleans are not integers. A Python bool is still an int (0 or 1), as in 3.4.0.
An explicit edge list declares a rectilinear dimension everywhere. ChunkGrid.from_sizes no longer collapses uniform edges, matching normalize_chunks_1d. So a stored RectilinearChunkGridMetadata passed as chunks= is treated the same whether its edges are uniform or not: zarr.create stores it as a rectilinear grid, as create_array does (3.4.0 stored RectilinearChunkGridMetadata(((5, 5),)) as a regular [5], A rectilinear chunk spec with a short trailing chunk is silently normalized to a regular grid, changing resize semantics #4272). A RectilinearChunkGridMetadata made only of bare integers declares no edge list, so it is the regular grid it describes (accepted for Zarr format 2 and as the chunk shape of a sharded array).
Legacy zarr.create(..., zarr_format=2) used chunks or chunk_shape, so a NumPy array such as chunks=np.array([5, 3]) failed with "truth value of an array is ambiguous". A small predicate (_v2_chunks_given) now decides whether chunks was given: falsy values (None, 0, [], (), False, np.int64(0), np.array(0), np.array([0])) still mean automatic chunking, exactly as in 3.4.0; a NumPy array with more than one element is always given.
Errors: a non-integer scalar specification (2.0, np.float64(2.0), a 0-d float array) raises the normalizer's own TypeError naming it (was "object has no len()" or "len() of unsized object").
Typing: normalize_chunks_nd(chunks: ChunksLike | None, ...); shard_spec uses the existing aliases instead of Any.

Not changed in this patch release: chunks=True still raises ValueError, and a bool inside a specification is still read as 1 or 0. Rejecting booleans with one TypeError is part of the 3.5.0 follow-up.

Compatibility with 3.4.0

In the 244-row behaviour table run against a real 3.4.0 install, this branch changes no result that 3.4.0 handled correctly. The rows it changes: zarr.create(chunks=np.array([5, 5]), zarr_format=2) and np.array([]) now work (3.4.0 raised a truth-value error with current NumPy); a uniform RectilinearChunkGridMetadata passed to zarr.create is stored as rectilinear (gh-4272). check_patch.py: OK, 0 undocumented changes, 0 warnings-only changes.

Tests

tests/test_unified_chunk_grid.py: one table for the legacy Zarr format 2 chunks argument (every falsy spelling auto-chunks, np.array(7) → (7, 7), np.array([5, 3]) → (5, 3)) and one error test for np.array([0, 0]); the classifier's tests are replaced by one test that the Zarr format 2 and sharding restrictions recognize every rectilinear specification (whichever dimension carries the list, uniform edges included) through create_array and zarr.create; the uniform-edge resize test covers edges given as lists and as metadata.
tests/test_chunk_grids.py: NumPy booleans are rejected by the normalizer; NumPy inputs join the existing normalizer tables.

Merge with #4334

The two PRs conflict in two small hunks; the resolution keeps both changes:

src/zarr/core/chunk_grids.py, resolve_outer_and_inner_chunks: keep this PR's single normalizer call and add fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334's unit, outer = normalize_chunks_nd(shard_spec, array_shape, unit=chunks.chunk_shape).
tests/test_unified_chunk_grid.py: keep both imports (import re from fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334, from functools import partial from this PR).

🤖 Generated with Claude Code

Chunk specifications were parsed by `normalize_chunks_nd`, but a separate duck-typed classifier, `_is_rectilinear_chunks`, ran on the raw input first at three sites to decide whether the spec was rectilinear. Two opinions on the same input is the shape of the bug in zarr-developers#4374, and they could disagree: a 0-d numpy array counted as rectilinear because it has `__iter__`. The classifier is gone. Each site normalizes first and asks the resulting `ChunkGrid` (`is_regular`); stored rectilinear metadata passed as `chunks=` counts as rectilinear even when its edges are uniform. The shard resolver sends regular and rectilinear shard specs through the same normalizer. Also fixed on the way: - The legacy v2 branch of `AsyncArray.create` tested `chunks or chunk_shape`, so `zarr.create(chunks=np.array([...]), zarr_format=2)` failed with "truth value of an array is ambiguous". It now uses the `is not None` form the v3 branch already had. - 0-d numpy arrays unwrap to their scalar in both normalizers instead of failing with "len() of unsized object". - A non-integer scalar spec (`2.0`, `np.float64`) raises the normalizer's own TypeError instead of "object has no len()". Assisted-by: ClaudeCode:claude-fable-5-1

Assisted-by: ClaudeCode:claude-fable-5-1

read-the-docs-community · 2026-09-18T16:58:03Z

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34778781 | 📁 Comparing 9016f32 against latest (1187a43)

🔍 Preview build

9 files changed · ± 9 modified

± Modified

read-the-docs-community · 2026-09-18T16:58:04Z

Documentation build overview

📚 zarr-indexing | 🛠️ Build #34778780 | 📁 Comparing 9016f32 against latest (1187a43)

🔍 Preview build

No files changed.

codecov · 2026-09-18T16:59:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.33%. Comparing base (d7686f6) to head (9016f32).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4376      +/-   ##
==========================================
- Coverage   94.37%   94.33%   -0.04%     
==========================================
  Files          93       93              
  Lines       13171    13167       -4     
==========================================
- Hits        12430    12421       -9     
- Misses        741      746       +5

Files with missing lines	Coverage Δ
src/zarr/core/array.py	`98.18% <100.00%> (+0.09%)`	⬆️
src/zarr/core/chunk_grids.py	`96.40% <100.00%> (-0.38%)`	⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`_chunk_int` is the normalizer's single integer rule: anything Python's integer protocol accepts (int, numpy integer scalars, 0-d integer arrays), except bool. It replaces the repeated numbers.Integral checks and the two 0-d ndarray unwraps in normalize_chunks_1d / normalize_chunks_nd. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ilinear rejection The numpy cases that passed before this PR are dropped; the 0-d array and float cases move into the normalizer's table and error tests. The legacy zarr.create Zarr format 2 path gets a truthiness test and its rectilinear rejection is now asserted with a message match. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…elopers#4376 changelog Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rid.from_sizes `ChunkGrid.from_sizes` collapsed uniform edge lists to `FixedDimension` while `normalize_chunks_1d` keeps them as `VaryingDimension`, so `init_array` needed an `isinstance(chunks, RectilinearChunkGridMetadata)` guard to keep uniform stored rectilinear grids under the Zarr format 2 and sharding restrictions. Both now agree that a list declares a rectilinear dimension, and the guard is gone: the normalized grid is the one judge. Also: `normalize_chunks_nd` and the shard spec are typed with `ChunksLike`; a bool chunk size is reported as not a chunk size; an integral float (`10.0`) is pinned as rejected; `None` reaching the normalizer gets the generic non-integer `TypeError`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…one error A bool, np.bool_, or boolean array anywhere in a chunk specification now raises the same TypeError from the normalizer's one integer test, instead of three different errors depending on spelling. Document that a RectilinearChunkGridMetadata of bare integers is read as a regular grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…alsy v2 chunks (patch release) Nothing in a patch release may reject a chunk specification that zarr 3.4.0 accepted. The normalizer's integer test reads a Python `bool` as the `int` it is again, so `chunks=(True, 5)` and a `True` edge are a chunk size of 1; `chunks=True` and `chunks=None` raise 3.4.0's `ValueError` again. Numpy booleans stay rejected, as they were. The legacy `zarr.create(..., zarr_format=2)` again reads a falsy `chunks` (0, [], False) as not given and chunks automatically; a numpy array is always taken as given, so its truth value is never tested. The `TypeError` for every boolean spelling returns in the next minor release. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…2 create as zarr 3.4.0 did The legacy `zarr.create(..., zarr_format=2)` took every numpy array as a given chunk specification, so `np.array(0)`, `np.array(False)` and `np.array([0])` raised instead of auto-chunking as in zarr 3.4.0. A numpy array with more than one element has no truth value and is always given; a shorter one is read by `.any()`, its truth value, which is False when empty, so `np.array([])` auto-chunks like `[]` (as 3.4.0 did with numpy 2.1). The two legacy v2 tests become one table. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… _chunk_int `operator.index` raises `TypeError` for everything the `SupportsIndex` check rejected, so the check was redundant (identical results over bool, numpy bools and integers, 0-d and 1-d arrays, float, str, bytes, None, list and a custom `__index__` class). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…iven Replace the inline conditional expression in `AsyncArray._create` with a small named predicate. Results are identical on every probed input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add `np.array(7)` -> `(7, 7)` to `test_legacy_create_v2_chunks` and an error test for `np.array([0, 0])`, killing the mutants that drop either half of `_v2_chunks_given`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

d-v-b added 2 commits September 18, 2026 18:54

docs: add changelog fragment for zarr-developers#4376

9d0f413

Assisted-by: ClaudeCode:claude-fable-5-1

github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes and removed needs release notes Automatically applied to PRs which haven't added release notes labels Sep 18, 2026

d-v-b mentioned this pull request Sep 18, 2026

test: sample the full rectilinear chunk grid declaration space #4377

Draft

d-v-b force-pushed the fix/single-chunk-normalizer branch from 22d828c to 9d0f413 Compare September 18, 2026 17:00

d-v-b mentioned this pull request Sep 19, 2026

fix(chunk-grids): one invariant for zero-length axes across model, clamps, and metadata #4334

Draft

d-v-b and others added 13 commits September 25, 2026 21:39

Merge remote-tracking branch 'origin/main' into loop/4376

5b44dfc

docs: note legacy v2 chunks=0/[] and bool chunk sizes in the zarr-dev…

626e227

…elopers#4376 changelog Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: describe the 4376 changes for a patch release

3fe6a83

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(chunk_grids): read bool chunk sizes as rows of the normalizer table

e1bdbfd

Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Sep 26, 2026

fix: read mixed regular/rectilinear chunk grids written by zarr 3.2.x #4375

Draft

fix(metadata)!: make chunk sizes strict in metadata constructors and chunk specifications #4431

Draft

Navigation Menu

Uh oh!

fix: make the chunk normalizer the one judge of a chunk specification - #4376

fix: make the chunk normalizer the one judge of a chunk specification#4376
d-v-b wants to merge 15 commits into
zarr-developers:mainfrom
d-v-b:fix/single-chunk-normalizer

d-v-b commented Sep 18, 2026 •

edited

Loading

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

d-v-b commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Compatibility with 3.4.0

Tests

Merge with #4334

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

read-the-docs-community Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

codecov Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

d-v-b commented Sep 18, 2026 •

edited

Loading

read-the-docs-community Bot commented Sep 18, 2026 •

edited

Loading

read-the-docs-community Bot commented Sep 18, 2026 •

edited

Loading

codecov Bot commented Sep 18, 2026 •

edited

Loading