Support irregular chunking - #332
asinghvi17 wants to merge 3 commits into
Conversation
Coverage Report for CI Build 35620748692Warning No base build found for commit Coverage: 89.279%Details
Uncovered Changes
Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
|
How is this related to #326? |
|
Was asking myself the same thing. #326 is written by a human, tests pass except that I am constantly rebasing against main as a moving target. |
|
Ok, I think your bot was definitely looking at my PR. strategy is the same as in my implementation, some names were cleaned up and code got a bit more verbose but looks ok IMO. Even the test example in the interop test is the same. |
|
I hadn't seen 326 when I had the bot make this, it was mostly for my own testing to make sure irregular chunks would fit my usecase. Can close once the subpackage PR lands and we rebase #326 |
1d133c8 to
34ffa9a
Compare
edbf5c1 to
33880e2
Compare
33880e2 to
134c122
Compare
Brings the rectilinear support to parity with #326. - A regular axis inside a rectilinear grid stores a full-size final chunk, as the spec and zarr-python do. It was clipped to the array, which neither side could read back. - Allow `resize!` and `append!` along the regular axes of a rectilinear array; irregular axes still reject before mutating. - Reject edge lists that overflow the array extent with a clear error instead of decoding the final chunk at the wrong size. - Count chunks from sizes so empty axes prune and index as on main. - `zinfo` reports the per-axis chunk sizes for rectilinear arrays. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38
A rectilinear grid depends on the array shape, which is already a Ref so that `resize!` can change it. Rebuilding the regular axes on every read left `metadata.chunks` stale and could not express a resized irregular axis. - `MetadataV3.chunks` is a `RefValue` holding an `NTuple` or `GridChunks`; `MetadataV2` is unchanged and shared code reads through `chunkspec`. - Growing an irregular axis appends one chunk covering the added extent, as zarr-python does. Shrinking one still throws before mutating, since zarr-python expresses that with overflowing edges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38
245d87b to
f06871e
Compare
Summary
Stacked on #342, which fixes
resize!on read-only arrays and consolidated stores; this PR's changes toresize!build on it.Relation to #326
Same strategy and user-facing API as #326 (
chunks=DiskArrays.GridChunks(...)forzcreate,zzeros, andZArray(data; chunks)), brought to parity with it:resize!/append!along regular axes of a rectilinear array@test_brokenzarr_format=2ArgumentErrormetadata.chunksRef{GridChunks}in v2 and v3Refholding anNTuple(regular) orGridChunks(rectilinear); v2 unchangedStored chunk extents were checked against zarr-python 3.3.0: it stores the final chunk of an integer axis at the full chunk size.
Known limitation
The spec lets a list of edge lengths overflow the array extent, and zarr-python leaves such a list behind when it shrinks a rectilinear array (it changes
shapeand keepschunk_shapes).DiskArrays.IrregularChunkscannot describe a chunk that extends past the array, so Zarr.jl throws anArgumentErrorwhen opening one. For the same reason Zarr.jl cannot shrink an irregular axis itself. Supporting both is left for a follow-up.Validation
On Julia 1.12 after the rebase:
🤖 Generated with Claude Code
https://claude.ai/code/session_01YY277jAB9XVg9KqrBuxL38