Skip to content

Forbid zero-length chunk references in ChunkManifest - #1089

Open
TomNicholas wants to merge 3 commits into
zarr-developers:mainfrom
TomNicholas:forbid-zero-length-chunks
Open

Forbid zero-length chunk references in ChunkManifest#1089
TomNicholas wants to merge 3 commits into
zarr-developers:mainfrom
TomNicholas:forbid-zero-length-chunks

Conversation

@TomNicholas

@TomNicholas TomNicholas commented Aug 17, 2026

Copy link
Copy Markdown
Member

Closes #1088.

A chunk must decode to the full chunk shape, so no valid chunk is ever zero bytes long — even an all-constant compressed chunk carries a codec header, and arrays with a zero-size dimension have no chunks at all. A zero-length reference is therefore always a bug, but nothing rejected it until read time, by which point it had already been committed to a store:

InvalidInputError: byte range From(0) is out of bounds for chunk of length 0

This PR instead validates at construction. ChunkManifest is the choke point every ManifestArray routes through, so the check covers all three constructors (ChunkManifest, ChunkManifest.from_arrays, ChunkEntry.with_validation) and both virtual and inlined references. A chunk which simply isn't stored keeps its existing spelling — the empty path — which reads back as fill_value and fetches nothing.

ValueError: Found a zero-length chunk reference to 'file:///…/sparse.tif' at index (0, 1).
A chunk must decode to the full chunk shape, so no valid chunk is ever zero bytes long. To
record that a chunk is not stored at all (as in a sparse array), give it the empty path ''
instead, which reads back as the array's fill_value without fetching anything.

Performance

from_arrays scans lengths first and only compares the paths array if a zero length is actually present — the StringDType comparison is ~30x more expensive than the uint64 scan. At 10M chunks with validate_paths=False (~0.3 ms before this PR), a manifest with no missing chunks costs 1.6 ms and one with any missing chunks ~52 ms.

The check is deliberately not behind validate_paths, which guards path validation specifically — the expensive np.vectorize loop. Gating it would mean bulk-generating parsers, exactly where this class of bug arises, are the ones that lose the guard.

Also here

conftest.py was generating scalar arrays with length: 0, because the scalar branch passes chunks=(0,) and np.prod((0,)) * itemsize == 0. A scalar array's single chunk holds one element, so that's a fixture bug — fixed, with the two affected assertions updated.

A chunk must decode to the full chunk shape, so no valid chunk is ever
zero bytes long. Such a reference is always a bug, but nothing rejected
it until read time - by which point it had already been committed to a
store.

Validate in ChunkManifest, which every ManifestArray construction path
routes through, covering virtual and inlined references via all three
constructors. A chunk which isn't stored keeps the existing spelling:
the empty path, which reads back as fill_value and fetches nothing.

In from_arrays the check scans lengths first and only compares the
paths array if a zero length is present, since the StringDType
comparison is ~30x more expensive than the uint64 scan. This keeps
from_arrays(validate_paths=False) at 1.6ms for 10M chunks.

Also fixes the test fixture generating scalar arrays with length 0,
which passed chunks=(0,) where a scalar's single chunk holds one
element.
argwhere materializes the position of every invalid chunk when only the
first is needed for the error message. On a manifest where most chunks
are invalid - the sparse COG case - that is a large allocation made
purely on the way to raising.
@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.79%. Comparing base (8dc2cb7) to head (ca2baa9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1089      +/-   ##
==========================================
+ Coverage   89.67%   89.79%   +0.12%     
==========================================
  Files          41       41              
  Lines        2682     2686       +4     
==========================================
+ Hits         2405     2412       +7     
+ Misses        277      274       -3     
Files with missing lines Coverage Δ
virtualizarr/manifests/manifest.py 94.15% <100.00%> (+0.15%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The 2D case subsumes the 1D one and additionally exercises unravel_index
with ndim > 1, so parametrize it over validate_paths and assert the
reported index in the same place.
@TomNicholas
TomNicholas marked this pull request as ready for review August 17, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Forbid zero-length chunk references in ChunkManifest

1 participant