Prototype prepared indexing execution for codec consumers - #323
Draft
d-v-b wants to merge 22 commits into
Draft
Conversation
Restricting a transform to a chunk box distributes over output dimensions whenever each output map reads its own input axis, which is every basic and orthogonal selection. Chunk resolution therefore no longer intersects the whole transform with every candidate chunk; it resolves each axis once against its grid into a table (StridedSet / IndexedSet), sorts correlated (vindex) index arrays into chunks once into a JointSet, and derives each ChunkProjection as one row of each table. ChunkPlan.partition() and partition_transform() expose the factored form, so a consumer can read the tables directly instead of materializing an object graph per chunk. The projections a plan yields are unchanged; the general whole-transform walk remains for hand-built diagonals, which have no factored form. Along the way: _intersect_general reuses a precomputed _CorrelatedBlock and accepts survivor positions; checked_affine has identity and dtype-bounded fast paths; ArrayMap._with_affine shares frozen index arrays on translate; IndexDomain._unchecked / IndexTransform._unchecked skip validation for objects derived from an already-valid transform. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
The package and its tests import nothing from zarr; the old comments claimed the chunk-resolution tests needed zarr's ChunkGrid, which stopped being true once the package grew its own grids. The real reason is the shared pinned test toolchain. Assisted-by: ClaudeCode:claude-fable-5-1
…hunk narrative The module docstring described intersecting the whole transform with every candidate chunk as "the algorithm"; that walk is now the fallback for hand-built diagonals only. It now explains the factored form and its three tables, and why they cost the sum of the touched chunks per axis. The visual guide gains a final integrator section, "A plan is a product of per-axis tables", with an executable snippet that reads the StridedSet, IndexedSet and JointSet tables off real plans and checks the plan's projections against the partition's rows. Integration boundaries gains "Reading the tables directly", a consumer that assembles a strided box from the tables with no projection materialized. The API index, landing page and design notes (TensorStore lineage, the performance caveat, and the box/query split) point at the new section. Assisted-by: ClaudeCode:claude-fable-5-1
…w fixes Adversarial review (roborev, a correctness reviewer, a complexity reviewer, and ~24k differential examples against main) of the grid partition. Cuts. The whole-transform walk that remained for hand-built diagonals is gone: it was unreachable for every index-array shape, its key builder was duplicated verbatim in _chunk_keys, and for the one shape it served it produced wrong projections (a three-point diagonal yielded four projections covering six cells, on main too). A DimensionMap diagonal is now rejected with ValueError. With it go the sorted-1-D fast path, the three cell-transform helpers, the block/positions parameters of _intersect_general, the correlated-residual check that admitted a diagonal and then crashed, GridPartition.__getitem__, partition_transform as public API, the object-dtype column fallback (StridedSet.origin is now a position along the request axis, so every column is intp), checked_affine's dtype-bound shortcut (measured at noise; the identity shortcut stays and now accepts bool via np.can_cast, as main did), and StridedSet.chunk_map/cell_map. Fixes. GridPartition.n_rows is an exact integer and len raises OverflowError instead of wrapping to zero; table columns are read-only, so a memoized partition cannot drift under a consumer; the documented table consumer now handles reversed axes, inserted axes and transposed transforms, and the snippet checks all three. Docs. Corrected the diagonal statement everywhere it appeared, the memoized "fresh walk" wording, the "vectorized per axis" claim, and the TensorStore correspondence (its strided sets are per input dimension; it keeps one index array set per connected component). The guide no longer restates the class docstrings. Assisted-by: ClaudeCode:claude-fable-5-1
A zero-stride DimensionMap over a domain wider than np.intp is valid and touches one storage cell; coercing every StridedSet column to intp made it raise OverflowError where main returned one projection. `extent` and `origin` are the two columns measured along the request axis, whose bounds are arbitrary Python ints, so they now fall back to exact-int (object) columns when a value does not fit. Chunk-local columns stay intp. Also corrects the design note that said both affine-diagonal cases raise NotImplementedError: two slice maps sharing an axis now raise ValueError. Assisted-by: ClaudeCode:claude-fable-5-1
…d the minimal grid protocol Every varying grid in the partition cases summed exactly to its extent, so the boundary where a chunk's data extent is shorter than its declared size, the rectilinear-specific case, was unpinned; so was a grid without data_size. Both now run through the evaluation oracle for strided, orthogonal and correlated selections. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
…cies Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Retain public selection-flow documentation and fix singleton data-extent coverage. Move the execution prototype to a follow-up review. Assisted-by: Codex:GPT-6
… columns, one planning mechanism Adversarial review of the branch (roborev, a correctness reviewer, a complexity reviewer, ~40k differential examples against main and NumPy). Fixes. Correlated planning probed storage bounds only through the grid's vectorized lookup, which zarr's grids do not validate, so an out-of-range coordinate silently planned a chunk that does not exist; the joint table now probes each component's extreme coordinates with the scalar lookup, as the orthogonal table already did. Table columns were read-only by flag only, which setflags(write=True) undoes; they are now re-homed over immutable bytes, like ArrayMap's index array, so a memoized partition cannot drift. Cuts, each with what it cost stated in the review: the affine-diagonal iteration path (a second mechanism that reintroduced transform.intersect per chunk for a shape no selection produces; diagonals now raise ValueError from iteration as from partition()), the single-component iterator (14-23% on a walk that stays 2-3x behind zarr's coordinate indexer either way), chunk_coord_batches with its beyond-intp mixed-radix path and n_rows (no consumer; a partition with row_shape writes the batching in three lines), and the stored block_coordinates column (byte-identical to positions on every 1-D block; now a memoized property). Tests that pinned memoization identity and column flags collapse into one that asserts immutability. Docs: the diagonal statement is now precise everywhere it appears, StridedSet.full no longer claims "in order", the guide no longer says sets holds every source axis, and the changelog describes the feature as shipped rather than the branch's history. Assisted-by: ClaudeCode:claude-fable-5-1
Use data-extent completeness, drop orthogonal scalar axes at the execution boundary, and run codec tests in the package CI suite. Keep ownership and write policies in this separate experimental review. Assisted-by: Codex:GPT-6
The planner now rejects diagonals with ValueError and has no shared projection walk, so execute_transform factors the plan up front and lets that rejection surface at construction instead of on first iteration. _axis_plan reuses chunk_resolution's data-extent helper rather than carrying a copy. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d-v-b
force-pushed
the
codex/indexing-execution
branch
from
September 6, 2026 10:07
1f2b4bc to
dc2e279
Compare
Merge current main and exercise arbitrary paired-coordinate reads and last-write-wins writes through NumPy and shard consumers. Keep optional codec integration imports compatible with lint. Assisted-by: Codex:GPT-6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
This draft adds an opt-in prepared execution layer on top of the grid planner now in main. It lets codec consumers obtain chunk-local and result selectors directly from affine runs or connected-component tables, while retaining declarative projection lowering for other transforms. It does not replace Zarr's default indexers.
Preparation makes snapshot/borrow ownership and read/write conflict policies explicit. NumPy and shard consumers lower the same work to their respective selector layouts. The selection-flow guide documents the base planner; this draft documents the experimental execution contracts and current sharding materialization boundary.
Review corrections included here:
No end-to-end storage speedup is claimed. Compact sharding consumption and broader dispatch remain future work.
The grid-planning prerequisite has landed and this branch now targets main. Review added a coordinate-gather read and last-write-wins property test. Validation: 1,473 package tests passed (4 existing skips), plus 40 real-codec integration tests; Ruff passed and Pyright had zero errors.