Skip to content

Support irregular chunking - #332

Open
asinghvi17 wants to merge 3 commits into
fix/resize-guardsfrom
feat/irregular-chunking
Open

asinghvi17 wants to merge 3 commits into
fix/resize-guardsfrom
feat/irregular-chunking

Conversation

@asinghvi17

@asinghvi17 asinghvi17 commented Aug 22, 2026 •

Copy link
Copy Markdown
Member

Summary

  • add Zarr v3 rectilinear chunk-grid metadata support
  • read and write arrays across irregular chunk boundaries
  • validate, encode, and decode inline chunk shapes, including run-length encoding
  • document irregular chunking and add Julia/Python interoperability coverage

Stacked on #342, which fixes resize! on read-only arrays and consolidated stores; this PR's changes to resize! build on it.

Relation to #326

Same strategy and user-facing API as #326 (chunks=DiskArrays.GridChunks(...) for zcreate, zzeros, and ZArray(data; chunks)), brought to parity with it:

Behaviour #326 this PR
Regular axis inside a rectilinear grid stores a full-size final chunk (what zarr-python writes) yes yes (was clipped before this update)
resize! / append! along regular axes of a rectilinear array grow only grow and shrink
Growing an irregular axis shape changes but the grid does not, so the metadata no longer validates appends one chunk covering the added extent, as zarr-python does
Shrinking an irregular axis throws after mutating the shape throws before mutating
Declared edges that overflow the array extent (allowed by the spec) rejected rejected with an explicit error
Julia → zarr-python data round trip @test_broken passes
Irregular grid with zarr_format=2 silently persists the max chunk size, so reopened data is wrong ArgumentError
metadata.chunks Ref{GridChunks} in v2 and v3 v3 only: Ref holding an NTuple (regular) or GridChunks (rectilinear); v2 unchanged

Stored chunk extents were checked against zarr-python 3.3.0: it stores the final chunk of an integer axis at the full chunk size.

Known limitation

The spec lets a list of edge lengths overflow the array extent, and zarr-python leaves such a list behind when it shrinks a rectilinear array (it changes shape and keeps chunk_shapes). DiskArrays.IrregularChunks cannot describe a chunk that extends past the array, so Zarr.jl throws an ArgumentError when opening one. For the same reason Zarr.jl cannot shrink an irregular axis itself. Supporting both is left for a follow-up.

Validation

On Julia 1.12 after the rebase:

  • full Julia test suite: 2,903 passed
  • Python interoperability in both directions, including a non-divisible regular axis and growing an irregular axis from either side
  • documentation build and doctests passed

🤖 Generated with Claude Code

https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38

@coveralls

coveralls commented Aug 22, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 35620748692

Warning

No base build found for commit eef5469 on fix/resize-guards.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 89.279%

Details

  • Patch coverage: 7 uncovered changes across 3 files (173 of 180 lines covered, 96.11%).

Uncovered Changes

File Changed Covered %
ZarrCore/src/ZArray.jl 97 93 95.88%
ZarrCore/src/metadata3.jl 69 67 97.1%
ZarrCore/src/metadata.jl 14 13 92.86%

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 2080
Covered Lines: 1857
Line Coverage: 89.28%
Coverage Strength: 2974.38 hits per line

💛 - Coveralls

@felixcremer

Copy link
Copy Markdown
Member

How is this related to #326?

@meggart

meggart commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Was asking myself the same thing. #326 is written by a human, tests pass except that I am constantly rebasing against main as a moving target.

@meggart

meggart commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Ok, I think your bot was definitely looking at my PR. strategy is the same as in my implementation, some names were cleaned up and code got a bit more verbose but looks ok IMO. Even the test example in the interop test is the same.

@asinghvi17

Copy link
Copy Markdown
Member Author

I hadn't seen 326 when I had the bot make this, it was mostly for my own testing to make sure irregular chunks would fit my usecase. Can close once the subpackage PR lands and we rebase #326

@asinghvi17
asinghvi17 force-pushed the feat/irregular-chunking branch from 1d133c8 to 34ffa9a Compare September 12, 2026 13:17
@asinghvi17
asinghvi17 changed the base branch from subpkg-split-clean to top-level-package-workspace September 12, 2026 13:17
@asinghvi17
asinghvi17 force-pushed the feat/irregular-chunking branch 2 times, most recently from edbf5c1 to 33880e2 Compare September 18, 2026 09:31
@asinghvi17
asinghvi17 changed the base branch from top-level-package-workspace to main September 18, 2026 09:31
@asinghvi17
asinghvi17 force-pushed the feat/irregular-chunking branch from 33880e2 to 134c122 Compare September 18, 2026 15:12
asinghvi17 and others added 3 commits September 19, 2026 05:38
Brings the rectilinear support to parity with #326.

- A regular axis inside a rectilinear grid stores a full-size final
  chunk, as the spec and zarr-python do. It was clipped to the array,
  which neither side could read back.
- Allow `resize!` and `append!` along the regular axes of a rectilinear
  array; irregular axes still reject before mutating.
- Reject edge lists that overflow the array extent with a clear error
  instead of decoding the final chunk at the wrong size.
- Count chunks from sizes so empty axes prune and index as on main.
- `zinfo` reports the per-axis chunk sizes for rectilinear arrays.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38
A rectilinear grid depends on the array shape, which is already a Ref so
that `resize!` can change it. Rebuilding the regular axes on every read
left `metadata.chunks` stale and could not express a resized irregular
axis.

- `MetadataV3.chunks` is a `RefValue` holding an `NTuple` or `GridChunks`;
  `MetadataV2` is unchanged and shared code reads through `chunkspec`.
- Growing an irregular axis appends one chunk covering the added extent,
  as zarr-python does. Shrinking one still throws before mutating, since
  zarr-python expresses that with overflowing edges.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38
@asinghvi17
asinghvi17 changed the base branch from main to fix/resize-guards September 19, 2026 09:55
@asinghvi17
asinghvi17 force-pushed the feat/irregular-chunking branch from 245d87b to f06871e Compare September 19, 2026 09:55
@asinghvi17
asinghvi17 added this pull request to stack #343 September 19, 2026 09:55
@asinghvi17
asinghvi17 marked this pull request as ready for review September 20, 2026 16:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants