Forbid zero-length chunk references in ChunkManifest - #1089
Open
TomNicholas wants to merge 3 commits into
Open
Conversation
A chunk must decode to the full chunk shape, so no valid chunk is ever zero bytes long. Such a reference is always a bug, but nothing rejected it until read time - by which point it had already been committed to a store. Validate in ChunkManifest, which every ManifestArray construction path routes through, covering virtual and inlined references via all three constructors. A chunk which isn't stored keeps the existing spelling: the empty path, which reads back as fill_value and fetches nothing. In from_arrays the check scans lengths first and only compares the paths array if a zero length is present, since the StringDType comparison is ~30x more expensive than the uint64 scan. This keeps from_arrays(validate_paths=False) at 1.6ms for 10M chunks. Also fixes the test fixture generating scalar arrays with length 0, which passed chunks=(0,) where a scalar's single chunk holds one element.
argwhere materializes the position of every invalid chunk when only the first is needed for the error message. On a manifest where most chunks are invalid - the sparse COG case - that is a large allocation made purely on the way to raising.
TomNicholas
temporarily deployed
to
test-release
August 17, 2026 12:05 — with
GitHub Actions
Inactive
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1089 +/- ##
==========================================
+ Coverage 89.67% 89.79% +0.12%
==========================================
Files 41 41
Lines 2682 2686 +4
==========================================
+ Hits 2405 2412 +7
+ Misses 277 274 -3
🚀 New features to boost your workflow:
|
The 2D case subsumes the 1D one and additionally exercises unravel_index with ndim > 1, so parametrize it over validate_paths and assert the reported index in the same place.
TomNicholas
marked this pull request as ready for review
August 17, 2026 12:09
TomNicholas
temporarily deployed
to
test-release
August 17, 2026 12:09 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1088.
A chunk must decode to the full chunk shape, so no valid chunk is ever zero bytes long — even an all-constant compressed chunk carries a codec header, and arrays with a zero-size dimension have no chunks at all. A zero-length reference is therefore always a bug, but nothing rejected it until read time, by which point it had already been committed to a store:
This PR instead validates at construction.
ChunkManifestis the choke point everyManifestArrayroutes through, so the check covers all three constructors (ChunkManifest,ChunkManifest.from_arrays,ChunkEntry.with_validation) and both virtual and inlined references. A chunk which simply isn't stored keeps its existing spelling — the empty path — which reads back asfill_valueand fetches nothing.Performance
from_arraysscanslengthsfirst and only compares thepathsarray if a zero length is actually present — the StringDType comparison is ~30x more expensive than the uint64 scan. At 10M chunks withvalidate_paths=False(~0.3 ms before this PR), a manifest with no missing chunks costs 1.6 ms and one with any missing chunks ~52 ms.The check is deliberately not behind
validate_paths, which guards path validation specifically — the expensivenp.vectorizeloop. Gating it would mean bulk-generating parsers, exactly where this class of bug arises, are the ones that lose the guard.Also here
conftest.pywas generating scalar arrays withlength: 0, because the scalar branch passeschunks=(0,)andnp.prod((0,)) * itemsize == 0. A scalar array's single chunk holds one element, so that's a fixture bug — fixed, with the two affected assertions updated.