Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support by erneestoc · Pull Request #2497 · TraceMachina/nativelink

erneestoc · 2026-07-02T20:55:40Z

Implements the server side of the REAPI blob split/splice extension (remote-apis#282) used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+ / 9.1.0+).

Fixes #2496.

What this adds

Vendored protocol: SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params, RepMaxCdcParams, and CacheCapabilities fields 8–12, verbatim from upstream remote-apis.
SpliceBlob re-assembles chunked uploads: validates chunk sizes/counts, checks all chunks exist (touching them), streams them through an incremental hasher into the CAS with reads pipelined 10-at-a-time — the digest is verified before EOF so a mismatch aborts the upload uncommitted — then persists the chunk layout in a configurable index store. The blob is fully materialized so ByteStream/FindMissingBlobs stay correct for non-chunking clients.
SplitBlob serves stored layouts (validated to sum to the blob size, so corrupt or truncated index entries are never served), and otherwise chunks blobs on demand with FastCDC 2020 — this is the path that matters for RBE, since worker-uploaded action outputs are never spliced by a client. Unusable layouts (evicted chunks, decode failures, transient existence-check errors) fall back to re-chunking.
Native proxy forwarding: grpc-store-backed instances forward both RPCs verbatim to the backend (instance-name rewriting + retries), making NativeLink relays transparent for chunking.
Capabilities advertise split_blob_support/splice_blob_support and FastCDC 2020 params per instance (collected across all server blocks, so split-listener deployments work), gated behind the new opt-in experimental_chunking CAS service config.
Observability: ChunkingMetrics — splice/split request totals, split hit/miss/on-demand counters, byte totals, and digest verification failures (tracked precisely via an explicit flag, not error-code inference).

Conformance

The chunker is the fastcdc crate at 3.2.1 — the exact version the REAPI spec names as a compliant implementation — used with the spec-mandated normalization level 2 (the crate default level 1 does not match). fastcdc_conformance_test.rs validates chunk offsets, lengths, SHA-256s, and gear-hash fingerprints against the official fastcdc2020_test_vectors.txt from remote-apis, for both seed 0 and seed 666, in both in-memory and streaming form. This is the guarantee that server-produced chunks dedupe byte-for-byte against Bazel-produced chunks. The canonical fixture image is the one already vendored at nativelink-util/tests/data/SekienAkashita.jpg for the existing FastCDC tests (its SHA-256 is asserted in the test).

Note: the vendored nativelink-util/src/fastcdc.rs used by DedupStore is not FastCDC-2020-compliant and is deliberately untouched — deployed dedup indexes depend on its exact boundaries; the two implementations coexist.

Safety / hardening

Digests are never trusted: splice verifies the re-assembled digest before commit; per-chunk content lengths are validated against their digests.
index_store == cas_store is rejected at startup (layouts stored under blob digests would overwrite blob content).
Setting index_store on a grpc-store instance is rejected (the backend owns layouts there).
Per-chunk size cap (16 MiB) and a configurable per-instance max_chunk_count (default 50k) bound memory, layout size, and response size; layout reads are capped consistently with the configured count and truncation is detected by the size-consistency check.
Off by default; with experimental_chunking unset, behavior is byte-for-byte identical to before (new RPCs return Unimplemented, capabilities advertise false).

Design decisions worth reviewer attention

Materialize-on-splice: the spliced blob is written whole into the CAS rather than serving reads from the layout. This keeps every existing read path correct and lets storage-level dedup compose naturally when cas_store is a DedupStore; the cost is chunks + blob both stored. A ChunkingStore that serves reads from layouts is a possible future direction.
Chunk/index lifetime coupling is best-effort: existence checks touch chunks, but stores answering from existence caches may not promote backing entries, and layouts can outlive chunks. Degradation is graceful (fallback to re-chunking or full download), never incorrect. A CompletenessCheckingStore-style wrapper is the analogous future fix.
ChunkingMetrics follows the ByteStreamMetrics pattern (not yet wired into a metrics root — same as existing service metrics).

Testing

23 service tests: splice/split round trip, digest & size mismatch rejection, missing-chunk NotFound, on-demand chunking (single- and multi-chunk with reassembly verification), layout reuse, unusable-layout fallback, absent-blob NotFound, disabled-instance Unimplemented, config rejection cases, and metric assertions throughout.
2 conformance tests against upstream vectors.
GrpcStore forwarding tested against a fake CAS backend over a real gRPC round trip (passthrough + instance-name rewriting).
bazel test green across nativelink-service, nativelink-store, nativelink-config, nativelink-util (63 tests) with clippy + rustfmt aspects; cargo fmt --check clean; bazel build //:nativelink succeeds.

Notes

Verified end-to-end against a real Bazel client (see PR comment for full numbers): incremental uploads of a ~30 MB artifact drop 99.4% for a single-point edit (183 KB) and 83.9% for six scattered edits (4.96 MB) — in line with published production CDC figures — with fetched bytes SHA-verified identical through both the ByteStream and SplitBlob paths.
Blobs exceeding the configurable max_chunk_count (default 50k, ~25 GiB at the default 512 KiB average) are served without chunking: SplitBlob returns NOT_FOUND and clients fall back to a regular download. Raise the knob for larger blobs, keeping client gRPC message limits in mind.
The chunking flag requires Bazel 8.7+/9.1+ on the client; the repo's current Bazel pin (9.0.2) predates it, so the LRE E2E jobs will start exercising this path once the pin moves past 9.1. Recommend 9.1.1+ specifically: 9.1.0 has an upstream client bug that corrupts outputs when --experimental_remote_cache_chunking is combined with --disk_cache (fixed in 9.1.1; reproducible against any chunking server).

🤖 Generated with Claude Code

This change is

Implements the server side of the remote-apis blob split/splice extension used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+/9.1.0+), fixes TraceMachina#2496. - Vendor SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params and CacheCapabilities fields 8-12 from upstream remote-apis. - SpliceBlob re-assembles chunked uploads: verifies chunk existence and the spliced digest before committing, materializes the blob so non-chunking clients stay correct, and persists the chunk layout in a configurable index store. Chunk reads are pipelined while hashing stays in chunk order. - SplitBlob serves stored layouts (validated against the blob size so corrupt or truncated index entries are never served), or chunks blobs on demand with FastCDC 2020 (fastcdc crate, normalization level 2) so outputs uploaded whole by remote execution workers also get chunked downloads. Unusable layouts fall back to re-chunking. - Capabilities advertise split/splice support and FastCDC 2020 parameters per instance, collected across all server blocks, gated behind the new opt-in experimental_chunking CAS service config (off by default; zero behavior change when unset). - Reject foot-gun configs at startup: index_store == cas_store (chunk layouts stored under blob digests would overwrite blob content) and chunking on grpc proxy stores (would download/re-upload entire blobs instead of forwarding RPCs). - Conformance-test the chunker against the official REAPI fastcdc2020_test_vectors.txt (offsets, lengths, sha256s and gear fingerprints, seeds 0 and 666, in-memory and streaming). - Add ChunkingMetrics (splice/split totals, hit/miss/on-demand rates, byte counters, digest verification failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

For grpc-store-backed CAS instances the chunking RPCs are now forwarded verbatim to the backend (with instance-name rewriting and the store's usual retry handling) instead of being rejected at startup. This makes NativeLink relays transparent for content-defined chunking: one RPC in, one RPC out, the backend owns chunking and the layout index. - Add GrpcStore::split_blob/splice_blob following the existing find_missing_blobs/batch_*/get_tree forwarding pattern. - Shortcut to the proxy in the CAS handlers before any local chunking machinery is consulted, mirroring the other four CAS RPCs. - Make experimental_chunking.index_store optional: required for locally chunked instances, rejected for grpc-store instances where the backend owns the chunk layouts. The capabilities service still advertises split/splice + FastCDC params from the same config block, so relay operators set avg_chunk_size_bytes to match their backend. - Test forwarding against a fake CAS backend over a real gRPC round trip (verifies passthrough and instance-name rewriting) and the new constructor rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel · 2026-07-02T20:55:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 7, 2026 6:21am
nativelink-aidm	Ready	Preview, Comment	Jul 7, 2026 6:21am

- Reuse the FastCDC test fixture already vendored at nativelink-util/tests/data/SekienAkashita.jpg (and already excluded from the forbid-binary-files hook) instead of adding a duplicate binary copy; export it from nativelink-util for the conformance test. - Format nativelink-service/Cargo.toml per taplo. - Replace the hardcoded 50k chunk cap with a per-instance experimental_chunking.max_chunk_count knob (default 50000). Blobs above the cap are served without chunking; the layout read cap is derived from the configured count so the two can never disagree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

MarcusSorealheis

One comment, really.

For new configuration options, we'd like to clarify that this is optional and include a configuration example. See ./nativelink-config and lots of examples.

Otherwise, this one is definitely on the correct path.

Address review feedback: state explicitly that experimental_chunking is optional (with unchanged behavior when unset) and add a complete, test-validated configuration example at nativelink-config/examples/chunking_cas.json5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

erneestoc · 2026-07-06T21:11:09Z

Thanks for the review! Addressed in ed98063:

Added a complete configuration example at nativelink-config/examples/chunking_cas.json5 (validated by the existing test_example_parsing config test), with each field annotated as optional/required and its default.
The experimental_chunking doc comment now states explicitly that the option is optional and that behavior is byte-for-byte unchanged when it is unset, and points to the example.

Resolves lockfile conflicts: Cargo.lock and MODULE.bazel.lock taken from main and regenerated by cargo/bazel to re-include the fastcdc dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

erneestoc · 2026-07-06T21:49:16Z

End-to-end verification against a real Bazel client (NativeLink from this branch + --experimental_remote_cache_chunking, wire bytes measured at the TCP socket with a byte-counting proxy). Each row: incremental re-upload after a small change, chunking off vs on.

Artifact (change)	Upload, chunking off	Upload, chunking on	savings
30 MB text, 1 edit at front (CDC best case)	30.7 MB	183 KB	99.4%
30 MB text, 6 scattered edits	30.7 MB	4.96 MB	83.9%
16.8 MB tar layer, 1 file changed	16.8 MB	2.1 MB	87.5%
3.4 MB clang-linked binary, 1 source file changed	3.4 MB	653 KB	80.8%
7.2 MB gzip of the tar (change near end of stream)	7.2 MB	3.3 MB	54.2%

Representative artifacts (tar layers, linked binaries, scattered edits) cluster at ~80–88%, matching BuildBuddy's published production figure of ~85% deduped bytes. Compressed artifacts are position-dependent: everything after the first changed byte re-uploads (a front-of-stream change would approach 0% savings) — inherent to CDC, not this implementation. Download side mirrors upload: a fresh client re-fetching the edited text artifact downloaded 178 KB instead of 30.7 MB.

Correctness verified end-to-end: fetched artifacts are byte-identical to expected (SHA-256) via both the plain ByteStream path (validates splice materialization) and the chunked SplitBlob path. Server-side stores show the expected mechanics (shared chunks + layout entries, no errors). Capability gating verified: with the server not advertising chunking, a flag-enabled client falls back to full uploads.

One upstream finding: Bazel 9.1.0's --experimental_remote_cache_chunking combined with --disk_cache reassembles corrupt outputs (reproducible against any server; the RPCs the server serves are identical and correct). Bazel 9.1.1 fixes it — worth recommending 9.1.1+/8.7+ in docs.

…lice

Now that the repo's Bazel is 9.1.1 (which supports --experimental_remote_cache_chunking), exercise the SplitBlob/SpliceBlob paths end-to-end in the existing integration-tests job: enable experimental_chunking in the docker-compose CAS config and add chunking_cache_test.sh, which uploads a ~6.9MB artifact as chunks, asserts the server registered a chunk layout, then re-fetches it through the chunked download path and verifies it is byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

MarcusSorealheis

We need the web/ documentation as well, as that's where most customers live as opposed to the repo.

Bazel 9.1.1 leaves digest_function unset in SplitBlob/SpliceBlob even when running with --digest_function=blake3 (surfaced by the new CI chunking integration test, which runs with the repo's blake3 default). REAPI length-based inference cannot disambiguate SHA256 from BLAKE3 (both 32 bytes), so: - SpliceBlob hashes the re-assembled blob with both candidates when the field is unset and accepts whichever reproduces the expected digest. - SplitBlob's on-demand chunking infers the blob's digest function with an extra content pass so chunk digests use the right function. Explicitly-set digest functions keep the single-hasher fast path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Address review: document the feature where customers look — what it is, measured savings, client requirements (Bazel 9.1.1+/8.7+), how to enable it, tuning knobs, and when it does or does not help. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Bazel has uploaded cache entries in the background by default since Bazel 8 (--remote_cache_async), so `bazel build` can return before the chunked upload and SpliceBlob complete. The CI logs show exactly this: the splice reached the server ~0.9s after "Build completed successfully", while the test asserted on the chunk index ~0.8s after the build returned. Forcing synchronous uploads makes both the chunk-layout assertion and the clean-rebuild cache-hit check deterministic. Verified by running the full script locally against a clean server: splice on upload, layout written, remote cache hit via SplitBlob on rebuild, SHA-identical output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The integration test harness exports NATIVELINK_DIR but launches the containers with `sudo env RUST_LOG=info docker compose up`, and sudo's env_reset strips the variable. Docker compose therefore fell back to mounting root's ~/.cache/nativelink instead of the per-run cache dir, so the per-test `find "$NATIVELINK_DIR" -delete` cleanup never touched real store state, and chunking_cache_test.sh asserted on a directory the CAS container never wrote to (its chunk layouts landed in /root/.cache/nativelink on the host). Pass the variable through sudo explicitly, and add diagnostics to the chunking test's failure path that list both candidate locations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

NativeLink expands only environment variables in config strings (shellexpand::env — the tilde feature is not enabled), so the "~/.cache/nativelink/..." paths in local-storage-cas.json5 were treated as literal relative paths: inside the container the CAS wrote to "/~/.cache/nativelink" under the process cwd, an unmounted ephemeral directory, never to the /root/.cache/nativelink bind mount. worker.json5 already uses absolute paths, which is why executor data was host-visible while CAS data was not. Use absolute paths to match, and drop the same misleading tilde idiom from the docs-site config snippet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

corcillo · 2026-07-07T01:28:03Z

Cool addition and impressive savings! @MarcusSorealheis ill leave it to you to approve since you had comments but i think theyve been addressed, namely via web/apps/docs/content/docs/configuration/chunking.mdx, and specifically highlights the 8.7/9.1.1 Bazel req as well.

…aceMachina#2497) * Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support Implements the server side of the remote-apis blob split/splice extension used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+/9.1.0+), fixes TraceMachina#2496. - Vendor SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params and CacheCapabilities fields 8-12 from upstream remote-apis. - SpliceBlob re-assembles chunked uploads: verifies chunk existence and the spliced digest before committing, materializes the blob so non-chunking clients stay correct, and persists the chunk layout in a configurable index store. Chunk reads are pipelined while hashing stays in chunk order. - SplitBlob serves stored layouts (validated against the blob size so corrupt or truncated index entries are never served), or chunks blobs on demand with FastCDC 2020 (fastcdc crate, normalization level 2) so outputs uploaded whole by remote execution workers also get chunked downloads. Unusable layouts fall back to re-chunking. - Capabilities advertise split/splice support and FastCDC 2020 parameters per instance, collected across all server blocks, gated behind the new opt-in experimental_chunking CAS service config (off by default; zero behavior change when unset). - Reject foot-gun configs at startup: index_store == cas_store (chunk layouts stored under blob digests would overwrite blob content) and chunking on grpc proxy stores (would download/re-upload entire blobs instead of forwarding RPCs). - Conformance-test the chunker against the official REAPI fastcdc2020_test_vectors.txt (offsets, lengths, sha256s and gear fingerprints, seeds 0 and 666, in-memory and streaming). - Add ChunkingMetrics (splice/split totals, hit/miss/on-demand rates, byte counters, digest verification failures). * Forward SplitBlob/SpliceBlob natively through grpc proxy stores For grpc-store-backed CAS instances the chunking RPCs are now forwarded verbatim to the backend (with instance-name rewriting and the store's usual retry handling) instead of being rejected at startup. This makes NativeLink relays transparent for content-defined chunking: one RPC in, one RPC out, the backend owns chunking and the layout index. - Add GrpcStore::split_blob/splice_blob following the existing find_missing_blobs/batch_*/get_tree forwarding pattern. - Shortcut to the proxy in the CAS handlers before any local chunking machinery is consulted, mirroring the other four CAS RPCs. - Make experimental_chunking.index_store optional: required for locally chunked instances, rejected for grpc-store instances where the backend owns the chunk layouts. The capabilities service still advertises split/splice + FastCDC params from the same config block, so relay operators set avg_chunk_size_bytes to match their backend. - Test forwarding against a fake CAS backend over a real gRPC round trip (verifies passthrough and instance-name rewriting) and the new constructor rules. * Fix pre-commit hooks and make max chunk count configurable - Reuse the FastCDC test fixture already vendored at nativelink-util/tests/data/SekienAkashita.jpg (and already excluded from the forbid-binary-files hook) instead of adding a duplicate binary copy; export it from nativelink-util for the conformance test. - Format nativelink-service/Cargo.toml per taplo. - Replace the hardcoded 50k chunk cap with a per-instance experimental_chunking.max_chunk_count knob (default 50000). Blobs above the cap are served without chunking; the layout read cap is derived from the configured count so the two can never disagree. * Add chunking configuration example and clarify optionality Address review feedback: state explicitly that experimental_chunking is optional (with unchanged behavior when unset) and add a complete, test-validated configuration example at nativelink-config/examples/chunking_cas.json5. * Format chunking example per formatjson5 * Add chunking integration test to CI Now that the repo's Bazel is 9.1.1 (which supports --experimental_remote_cache_chunking), exercise the SplitBlob/SpliceBlob paths end-to-end in the existing integration-tests job: enable experimental_chunking in the docker-compose CAS config and add chunking_cache_test.sh, which uploads a ~6.9MB artifact as chunks, asserts the server registered a chunk layout, then re-fetches it through the chunked download path and verifies it is byte-identical. * Infer digest function when unset in chunking RPCs Bazel 9.1.1 leaves digest_function unset in SplitBlob/SpliceBlob even when running with --digest_function=blake3 (surfaced by the new CI chunking integration test, which runs with the repo's blake3 default). REAPI length-based inference cannot disambiguate SHA256 from BLAKE3 (both 32 bytes), so: - SpliceBlob hashes the re-assembled blob with both candidates when the field is unset and accepts whichever reproduces the expected digest. - SplitBlob's on-demand chunking infers the blob's digest function with an extra content pass so chunk digests use the right function. Explicitly-set digest functions keep the single-hasher fast path. * Fix artifact path resolution in chunking integration test The test runner's working directory is deployment-examples/docker-compose, not the workspace root, so the bazel-bin convenience symlink is not at the script's cwd. Resolve the output path through bazel info instead. * Add docs-site page for content-defined chunking Address review: document the feature where customers look — what it is, measured savings, client requirements (Bazel 9.1.1+/8.7+), how to enable it, tuning knobs, and when it does or does not help. * Disable async cache uploads in chunking integration test Bazel has uploaded cache entries in the background by default since Bazel 8 (--remote_cache_async), so `bazel build` can return before the chunked upload and SpliceBlob complete. The CI logs show exactly this: the splice reached the server ~0.9s after "Build completed successfully", while the test asserted on the chunk index ~0.8s after the build returned. Forcing synchronous uploads makes both the chunk-layout assertion and the clean-rebuild cache-hit check deterministic. Verified by running the full script locally against a clean server: splice on upload, layout written, remote cache hit via SplitBlob on rebuild, SHA-identical output. * Pass NATIVELINK_DIR through sudo to docker compose in test harness The integration test harness exports NATIVELINK_DIR but launches the containers with `sudo env RUST_LOG=info docker compose up`, and sudo's env_reset strips the variable. Docker compose therefore fell back to mounting root's ~/.cache/nativelink instead of the per-run cache dir, so the per-test `find "$NATIVELINK_DIR" -delete` cleanup never touched real store state, and chunking_cache_test.sh asserted on a directory the CAS container never wrote to (its chunk layouts landed in /root/.cache/nativelink on the host). Pass the variable through sudo explicitly, and add diagnostics to the chunking test's failure path that list both candidate locations. * Use absolute store paths in docker-compose CAS config NativeLink expands only environment variables in config strings (shellexpand::env — the tilde feature is not enabled), so the "~/.cache/nativelink/..." paths in local-storage-cas.json5 were treated as literal relative paths: inside the container the CAS wrote to "/~/.cache/nativelink" under the process cwd, an unmounted ephemeral directory, never to the /root/.cache/nativelink bind mount. worker.json5 already uses absolute paths, which is why executor data was host-visible while CAS data was not. Use absolute paths to match, and drop the same misleading tilde idiom from the docs-site config snippet.

Merge tag 'v1.6.1' (d0b7a61) from TraceMachina/nativelink into our production fork. Resolves 90 conflicts across all crates; full workspace compiles with --features quic (0 errors). Key resolution decisions (see .claude/merge-v1.6.1-tracking.md): - Error model: kept our general Error.details: Vec<Any> superset (missing-blob PreconditionFailure + BackpressureSignal + WatchdogTimeoutSignal); dropped upstream's narrower ErrorContext enum; re-added additive From<base64::DecodeError>/From<NulError> impls. - Eviction: kept our MokaEvictingMap architecture; upstream's legacy EvictingMap struct + RemoveItemCallback dropped (our unified ItemCallback supersedes). - Trait cascade: adopted upstream update->Result<u64>, required post_init (kept our LAZY RefStore; did not wire eager run_post_init), SchedulerStore update_data expiry param (threaded None; kept our max_seconds retain TTL). - Serialization: adopted upstream bincode->wincode for dedup/compression (not in live prod chain, no on-disk data). Kept bincode for util locality map. - Deps: kept our tonic 0.14.5+tls-aws-lc, hyper fork, tokio-epoll-uring; took newer prost 0.14.4 / tokio 1.52 / opentelemetry 0.32. - The 4 most-diverged CAS files (fast_slow/filesystem/grpc/redis, 2600-8500 lines diverged) kept as ours + trait-delta; upstream's ~11 bug fixes audited per-fix (6 already-covered, 3 N/A-by-design, 2 ported); 2 inert config knobs (use_legacy_resource_names, bypass_dedup_threshold_bytes) wired. Deferred additive features (config-gated off, follow-up passes): - TraceMachina#2497 SplitBlob/SpliceBlob REAPI chunking: proto + capabilities wired (advertises only if configured), cas_server handlers stubbed unimplemented. - TraceMachina#2527 wire-compression: capabilities can advertise it (empty by default), bytestream serving not implemented. Declined (duplicate/unused/conflicting): redis Sentinel (unused), worker .exec variants / namespace_utils / persistent_worker / qos (Linux/arch features we don't use), concurrent dir materialization (TraceMachina#2526). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Follow-up to the merge commit, applying the 10-reviewer panel's fix batch (all findings default-safe; no BLOCK). Operator-approved. - Remove GrpcSpec.headers/forward_headers: declined upstream TraceMachina#2288 (unconsumed but doc-advertised as JWT/auth forwarding — misleading). - Capabilities: hardcode split_blob_support/splice_blob_support=false and supported_compressors=empty until real handler passes land (TraceMachina#2497 / TraceMachina#2527); handlers are Status::unimplemented, so advertising off a config knob was a half-wire. - Port regression tests for the two bucket-E fixes that shipped untested: TraceMachina#2424 filesystem unref-when-temp-dir-missing (no content orphan) and TraceMachina#2353 redis subscription drop-race (subscribed_keys lock-before-drop). Both mutation-verified. - Correct store-backed-scheduler retention comment (max_seconds is memory-backend-only; store/redis path has no completed-action TTL post-merge — expiry threaded None). Deferred task filed. - Harden filesystem get_file_entry_for_digest to return NotFound for zero-digest (matches batch variant; closes TraceMachina#2346 latent footgun). - Add // CAPPED annotations on the bounded origin-event senders. - Document: post_init intentionally never root-driven (lazy RefStore); resource_usage None (worker telemetry declined); the 3 unwired upstream subsystems (persistent_worker/namespace_utils/qos) kept inert. Retry/timeout semantics change (TraceMachina#1906 disconnect-counts-against-retries + per-action timeout deadline) adopted from upstream per operator decision; watch retry metrics post-deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…hunking (FastCDC-2020) Completes the runtime half of upstream TraceMachina#2497 (commit ed2ad12) onto the v1.6.1 merge. The base merge already vendored the config (experimental_chunking / CasChunkingConfig), regenerated proto (SplitBlob/SpliceBlob/FastCdc2020Params), the fastcdc v3.2.1 + tokio-util deps, digest_hasher::digest_hasher_func_from_context, verify_store's use of it, the fastcdc conformance test + REAPI vectors, the Bazel BUILD entries, the chunking_cas.json5 example, and the integration scaffolding — but left the cas_server SplitBlob/SpliceBlob handlers as Status::unimplemented stubs, hardcoded capabilities split/splice support to false, and dropped the handler + grpc-forward tests. This commit implements the real handlers. Invariant being violated: a CAS instance that opts into experimental_chunking must serve REAPI SplitBlob/SpliceBlob with byte-exact FastCDC-2020 chunking and advertise that support; the base merge left it unimplemented/hardcoded-off. Mechanism that violates it: cas_server.rs split_blob/splice_blob returned Status::unimplemented, and capabilities_server.rs hardcoded split_blob_support/splice_blob_support = false regardless of config. Mechanism that re-establishes it after the fix: cas_server.rs real inner_split_blob/inner_splice_blob/chunk_blob_on_demand wired to the instance's cas_store chain + configured index_store (built in CasServer::new with the same-store and grpc-store foot-gun rejections); grpc_store.rs split_blob/splice_blob forward the RPCs verbatim for grpc-backed instances; capabilities_server.rs advertises split/splice + FastCDC params gated on chunking_params.is_some() (populated iff experimental_chunking is set). Composite invariants this fix interacts with: - FL-688 ack-gate / WorkerProxyStore CAS chain: chunking is disabled in the production config (experimental_chunking: None), so chunking_instances is empty and every chunking store op is short-circuited by chunking_instance() returning Unimplemented — zero interaction with the ack-gate / mirror path in production. - No per-RPC timeouts (operator directive): the grpc split/splice forwarders follow the find_missing_blobs pattern with NO per-RPC deadline; the local handlers add none. Liveness stays on transport keepalive. - Bounded network buffers: per-chunk size capped at MAX_SPLICE_CHUNK_SIZE (16 MiB), in-flight chunk reads/writes bounded at CHUNK_CONCURRENCY (10), chunk_digests / SplitBlobResponse bounded by the per-instance max_chunk_count, layout reads capped at max_layout_size(); no fsync. Test that proves the fix re-establishes the invariant: fastcdc2020_matches_reapi_test_vectors (byte-exact vs REAPI vectors), splice_and_split_round_trip, and chunking_enabled_instance_advertises_split_splice_and_fastcdc_params. FastCDC reconciliation: TraceMachina#2497 uses the EXTERNAL fastcdc crate v3.2.1 v2020 module (AsyncStreamCDC / FastCDC, Normalization::Level2), which is distinct from our homegrown nativelink-util/src/fastcdc.rs (the dedup_store chunker). They coexist; our fastcdc.rs is untouched. Behavior changes (enumerated against production composition): - Production CAS instances (experimental_chunking: None): split_blob/ splice_blob still return Code::Unimplemented (via chunking_instance()), and capabilities still advertise split/splice = false. Only the Unimplemented error MESSAGE text changes ("Blob chunking is not enabled for instance '<name>'" vs the old stub string); the Code is unchanged and conformant clients never call the RPCs (support advertised false). No chunking store op is reachable in production. - Opt-in instances (experimental_chunking set, non-grpc cas_store): split_blob/splice_blob now serve real chunk layouts / re-assemble blobs; capabilities advertise support + FastCDC params. New behavior, off by default. - grpc-store-backed CAS instances with experimental_chunking (no index_store): split_blob/splice_blob now forward to the backend instead of Unimplemented. Not a production topology (prod cas_store is FilesystemStore via FastSlow). Verified: fastcdc conformance 2/2 byte-exact; cas_server_test 28/28 (12 new chunking); capabilities chunking 2/2; grpc_store_test 17/17 (1 new forward); workspace build --features quic green. 5 mutation-verify cycles (capabilities gate, splice digest verify, max_chunk_count, FastCDC normalization level, grpc instance-name rewrite) each fail with the test's bespoke assertion and restore green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

These test targets never compiled under `chunked_fast_slow[,test-utils]`; the breaks pre-date and are unrelated to the v1-chunked-path removal (two review cadres confirmed byte-identical base<->HEAD for the removal). Test-only; no production code changed. Fixes (all verified by compiling first, then re-running green): 1. Missing `bypass_dedup_threshold_bytes` field (added by TraceMachina#2497 FastCDC) in one `FastSlowSpec` literal: - fast_slow_store_334_backpressure_preservation_test.rs:170 (E0063) -> add `bypass_dedup_threshold_bytes: 0` (u64 default, matching every other literal in-tree). (The dispatch also named fast_slow_str_key_...; that file already carried the field at HEAD -- its real break was TraceMachina#2 below.) 2. `StoreLike::update` returns `Result<u64, Error>` but five test helpers still declare `Result<(), Error>` (E0308). Discard the byte count at the terminal expression (`.map(|_| ())`) -- least-invasive, preserves each helper's declared signature and every caller: - store/fast_slow_str_key_skips_chunked_dispatch_test.rs:142,417 - store/fast_slow_store_334_backpressure_preservation_test.rs:153 - service/chunked_stable_digests_push_test.rs:237 - service/bazel_facing_internal_chunking_test.rs:232 - service/chunked_p25_p27_e2e_test.rs:193 3. `chunked_commit_soft_warn_test` had no `[[test]]` block, so a whole-suite `cargo test --features chunked_fast_slow` (no test-utils) tried to compile it (its body is `#![cfg(feature = "chunked_fast_slow")]`) against test-utils-gated items instead of auto-skipping (E0432/E0599 x15). Add the entry with `required-features = ["chunked_fast_slow", "test-utils"]`, matching siblings. 4. (NOT in dispatch, found by compiling) chunked_b1_writev_test.rs:66 (E0425): the `skip_if_no_io_uring` guard -- explicitly documented to skip "when the io-uring feature is compiled out" -- called `nativelink_util::fs:: is_io_uring_available()`, which only exists under `#[cfg(all(feature = "io-uring", target_os = "linux"))]`, so the guard itself could not compile without io-uring. cfg-guard the call and default `available = false` on the complement, honoring the guard's documented intent. Invariant being violated: every declared `#[[test]]`/helper under `chunked_fast_slow[,test-utils]` must compile in the feature configs its `required-features` (or file-level cfg) permit. Mechanism that violates it: three drifted APIs -- TraceMachina#2497's new FastSlowSpec field, `StoreLike::update`'s `Result<u64>` return, and the io-uring-gated fs fn -- plus a missing Cargo `[[test]]` gate, left seven test files uncompilable. Mechanism that re-establishes it: add the field/`.map(|_| ())`/cfg-guard/`[[test]]` entry so each target compiles (and auto-skips) in exactly the configs it declares. Composite invariants this fix interacts with: none -- no runtime path, store chain, ack/pin/eviction semantics, or buffering is touched. Test that proves the fix: the targets themselves now compile and run green (fast_slow_str_key... 3 ok, fast_slow_store_334... 10 ok, chunked_p25_p27... 2 ok, chunked_stable_digests... 14 ok; bazel_facing... 20 ok + 1 pre-existing tracing-test global-buffer isolation flake, passes in isolation, unrelated). Behavior changes: none. All edits are test-only (test bodies, a test helper's return-value discard, a test's cfg guard, and a Cargo `[[test]]` declaration). No production source, config, or runtime path changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit d0df7641f18e08158ef44736712ef8a4f6e0aec9)

…g field The v1 WriteChunked worker-upload path was deleted; workers always use v2 (`update_via_chunked_inner` dispatches WriteChunkedV2 unconditionally). The `GrpcSpec.chunked_v2_writes_enabled` flag was parsed-but-ignored after that removal -- no runtime path read it. Remove the field, its docs, every struct literal that set it, and the stale comments that named it. DEPLOYED-CONFIG ORDERING: `GrpcSpec` is `#[serde(deny_unknown_fields)]`, so a config that STILL carries `chunked_v2_writes_enabled` is now a HARD parse error (worker fails to boot). Deployed worker configs carry `chunked_v2_writes_enabled: true` and MUST be stripped of the key BEFORE this build ships. The parent owns that config migration + deploy ordering. Invariant walk: - Invariant violated: the config schema must expose only knobs a runtime path honors -- a parsed-but-ignored field implies a v1/v2 write-path selector that no longer exists, misleading operators and reviewers. - Mechanism that violates it: `GrpcSpec.chunked_v2_writes_enabled` (nativelink-config/src/stores.rs:1847, pre-removal) was deserialized but never read after the v1 dispatcher was deleted. - Mechanism that re-establishes it: delete the field from `GrpcSpec` and every struct literal; `deny_unknown_fields` (stores.rs:1659) now rejects any lingering key, forcing operators to remove the dead knob. - Composite invariants this interacts with: deny_unknown_fields config-parse safety (deployed configs must be stripped first -- parent-owned deploy ordering); the sibling `chunked_writes_enabled` field is retained and unchanged; the v2 dispatch path is untouched (already unconditional). - Proving tests: grpc_spec_rejects_removed_chunked_v2_writes_enabled_key (deny-unknown-field reject contract) and grpc_spec_parses_without_chunked_v2_writes_enabled_key (absence-OK). Behavior changes (each vs production composition): - nativelink-config GrpcSpec deserialization: a config setting `chunked_v2_writes_enabled` now FAILS to parse (was: accepted-and-ignored). Production composition: live worker configs set it `true`, so they must be stripped of the key before this ships or the worker will not boot. This is the ONLY runtime behavior change; it is intentional and the reason for the deploy-ordering note above. - Write path: NONE. `update_via_chunked_inner` already dispatched v2 unconditionally before this change (the field was already ignored), so the data plane is unaffected. - Test/bench struct-literal removals + comment rewords: no runtime behavior. Verification: - cargo check --bin nativelink --release --features quic,pprof,chunked_fast_slow -> clean (only a pre-existing unrelated tokio_unstable cfg warning). - cargo test -p nativelink-config -- grpc_spec -> new reject/absence tests pass; mutation (re-add field) -> reject test fails with its bespoke message -> restored -> green. - git grep -w chunked_v2_writes_enabled: zero struct-literal / field-read refs; only the intentional reject-test key string + doc comments remain. Pre-existing (NOT introduced here): nativelink-worker test `backfill_is_worker_header_seam_test` fails to compile at base d0df7641 -- its `impl ContentAddressableStorage` mock lacks `split_blob`/`splice_blob` added by 63a271f5 (TraceMachina#2497). The diff to that file here is the single field-line removal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit f700ac1a8bbf8101cc2cc415c35c47cb891112b4)

erneestoc and others added 2 commits July 2, 2026 13:41

vercel Bot deployed to Preview – nativelink-aidm July 2, 2026 20:56 View deployment

vercel Bot deployed to Preview – nativelink July 2, 2026 20:56 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 2, 2026 22:39 View deployment

vercel Bot deployed to Preview – nativelink July 2, 2026 22:40 View deployment

MarcusSorealheis reviewed Jul 3, 2026

View reviewed changes

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:11 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:12 View deployment

Merge branch 'main' into experimental-cdc-split-splice

a03db5c

Resolves lockfile conflicts: Cargo.lock and MODULE.bazel.lock taken from main and regenerated by cargo/bazel to re-include the fastcdc dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:16 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:17 View deployment

Format chunking example per formatjson5

a65ed71

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:40 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:41 View deployment

erneestoc requested a review from MarcusSorealheis July 6, 2026 21:56

erneestoc marked this pull request as ready for review July 6, 2026 22:51

erneestoc and others added 2 commits July 6, 2026 16:37

Merge remote-tracking branch 'tm/main' into experimental-cdc-split-sp…

0942bca

…lice

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 23:40 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 23:41 View deployment

MarcusSorealheis requested changes Jul 6, 2026

View reviewed changes

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 23:56 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 23:56 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 00:12 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 00:13 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 00:17 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 00:17 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 00:36 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 00:37 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 00:50 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 00:50 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 01:04 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 01:05 View deployment

MarcusSorealheis approved these changes Jul 7, 2026

View reviewed changes

Merge branch 'main' into experimental-cdc-split-splice

1c9ed08

vercel Bot deployed to Preview – nativelink-aidm July 7, 2026 06:20 View deployment

vercel Bot deployed to Preview – nativelink July 7, 2026 06:21 View deployment

palfrey merged commit 9d86aaf into TraceMachina:main Jul 7, 2026
50 checks passed

erneestoc mentioned this pull request Jul 7, 2026

Merge TraceMachina main (REAPI chunking) to unblock #2416 Reactor-Inc/nativelink#1

Closed

erneestoc deleted the experimental-cdc-split-splice branch July 11, 2026 00:57

This was referenced Jul 22, 2026

Add opt-in CDC chunked uploads to GrpcStore #2595

Open

Demote CDC chunks in the eviction order after splice #2627

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support - #2497

Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support#2497
palfrey merged 15 commits into
TraceMachina:mainfrom
erneestoc:experimental-cdc-split-splice

erneestoc commented Jul 2, 2026 •

edited by MarcusSorealheis

Loading

Uh oh!

vercel Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

MarcusSorealheis left a comment •

edited

Loading

Uh oh!

erneestoc commented Jul 6, 2026

Uh oh!

erneestoc commented Jul 6, 2026 •

edited

Loading

Uh oh!

MarcusSorealheis left a comment

Uh oh!

corcillo commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

erneestoc commented Jul 2, 2026 • edited by MarcusSorealheis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Conformance

Safety / hardening

Design decisions worth reviewer attention

Testing

Notes

Uh oh!

vercel Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcusSorealheis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erneestoc commented Jul 6, 2026

Uh oh!

erneestoc commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcusSorealheis left a comment

Choose a reason for hiding this comment

Uh oh!

corcillo commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

erneestoc commented Jul 2, 2026 •

edited by MarcusSorealheis

Loading

vercel Bot commented Jul 2, 2026 •

edited

Loading

MarcusSorealheis left a comment •

edited

Loading

erneestoc commented Jul 6, 2026 •

edited

Loading