Add zstd at-rest CAS compression with byte-for-byte wire passthrough

[walter-zeromatter]

Problem

NativeLink accepts and advertises REAPI compressed-blobs/zstd (#2527), and since #2596 it also sends it for its own GrpcStore hops. Both undo the compression at the store boundary: the CAS holds raw bytes, so every --remote_cache_compression client pays a decompress-on-write and a recompress-on-read even though the bytes were zstd on the wire the whole way. On a cache hit the server decompresses stored bytes only to immediately re-encode them at the gRPC boundary.

Mechanism

A new zstd algorithm for the existing compression store keeps CAS blobs as standard zstd frames at rest. When it is the store an instance points at directly, it advertises a WireCompressionStore capability that ByteStream read/write and BatchReadBlobs/BatchUpdateBlobs use to move the stored stream byte-for-byte, with no decode/re-encode round trip. Identity clients (no --remote_cache_compression) are unaffected — they still get plain decompressed bytes, encoded and decoded on the way in and out.

The capability is deliberately not forwarded through wrappers (Store::wire_compression_store does not follow inner_store): a wrapper outside a representation-changing store cannot promise its bytes are a valid wire stream.

Uploads are validated before anything is committed. A client-supplied compressed stream is decoded to recompute the length and hash, checked against the declared digest, and only then handed to the backend — so a bad stream can never reach the inner store.

Deployment constraints

These are real and do not apply to compression_algorithm: { lz4: ... }:

• CAS-only. Every operation needs a digest key. Never point an AC store at it.
• backend must be a dedicated, empty namespace. It stores digest → zstd bytes under the same digest keys, and raw CAS payloads are unstructured (an arbitrary blob can legitimately begin with the zstd magic number), so there is no safe way to sniff which encoding an entry is in. Rollout and rollback need a cache flush or a new namespace; there is no in-place migration.
• Placement matters. It must be the store the instance points at directly for passthrough to apply. fast_slow, dedup, existence_cache, cache_metrics, shard, ref_store, and size_partitioning all compose fine inside it. A wrapper outside it stays correct but disables passthrough at that boundary.
• compression_level is capped at 1..=19, not zstd's 22. Levels 1–19 cap the frame at windowLog ≤ 23 (≤ 8 MiB), decodable by every Bazel/zstd-jni client on plain libzstd defaults. Long-distance matching and dictionaries are never enabled.

deployment-examples/docker-compose/local-storage-cas-zstd.json5 is a complete annotated deployment, and /reference/nativelink-config/store-overview#pass-through-compression-compression_algorithmzstd documents the whole picture.

Bounds and admission

Compressed input is bounded from both ends. max_compressed_upload_size caps the compressed bytes a client may push (RESOURCE_EXHAUSTED); independently the decoder's output sink is capped at the digest's declared uncompressed size and rejects with INVALID_ARGUMENT the moment decoded bytes would exceed it, so a zstd bomb dies at the first over-limit block instead of being materialized. Zero-digest validation runs through the same bounded decoder with an output cap of zero.

Three deadlines and two admission bounds keep a stalled or hostile peer from parking on resources:

• stage_timeout_s (default 600) — total validate-and-stage budget, measured from admission. Unlike a per-message idle timeout, continuous slow progress does not reset it, so a client trickling bytes cannot hold a staging slot open.
• commit_timeout_s (default 300) — bounds optional recompression plus the inner-store commit.
• compressed_upload_idle_timeout_s (default 60, a ByteStream instance setting) — bounds the wait for the next WriteRequest.
• max_concurrent_staged_uploads (default 4) and max_concurrent_identity_ops (default 256) — the latter because identity transfers hold a blocking thread for their whole duration.

Every path — success, validation failure, any timeout, cancellation — removes the staged temp file and releases the permit.

Two throughput notes: compressed uploads at or below max_inline_commit_size (default 4 MiB) are validated and committed straight from memory with no staging file and no fsync, because BatchUpdateBlobs payloads are small and numerous. And recompression is best-effort — an upload that finds every recompression slot busy commits its original stream rather than queuing while holding a staging permit.

Interaction with #2596

#2596 is merged, and this branch is rebased on top of it. They compose, with one caveat worth documenting:

• ✅ Add opt-in zstd wire compression to GrpcStore transfers #2596 encodes client-side at WIRE_COMPRESSION_ZSTD_LEVEL = 1 (deliberately cheap) for blobs at or above 64 KiB. A server-side compression_algorithm.zstd with a higher compression_level and max_recompression_size > 0 re-encodes exactly those uploads and keeps whichever stream is smaller — cheap client encode, better ratio at rest.
• ⚠️ Do not enable GrpcSpec.experimental_remote_cache_compression on a grpc store used as the backend of a zstd compression store. The backend would receive bytes that are already zstd and encode them again for the wire. Symmetric, so not a correctness problem, but pure CPU cost on both ends.

Tests

nativelink-store/tests/zstd_store_test.rs — 49 tests, plus 6 unit tests in zstd_store.rs itself: identity round trips, ranged reads that must drain the inner stream, byte-for-byte passthrough including concatenated frames, digest/size mismatch rejection with non-commitment, incomplete trailing frames, oversize rejection, zstd-bomb rejection at the sink, zero-digest handling, staging-semaphore serialization, stage/commit timeouts releasing their slot, temp-file cleanup on cancellation, inline vs staged commit, a filesystem-backend commit (which commits by rename(2) rather than from the descriptor), and composition through fast_slow / dedup / existence_cache / cache_metrics / size_partitioning / shard / ref_store, plus a wrapper-outside case.

Service-level coverage in bytestream_server_test.rs and cas_server_test.rs exercises both fast paths end-to-end, including the compressed-upload idle timeout freeing a staging slot and a continuously-progressing upload not being timed out.

Verified on this branch:

bazel test //...   # 103/103 pass, with the clippy pedantic/nursery + rustfmt aspects
cargo test -p nativelink-store -p nativelink-service -p nativelink-config -p nativelink-util  # 72 suites, 0 failures
cargo clippy --all-targets -p nativelink-store -p nativelink-service -p nativelink-config     # clean
cargo +nightly fmt --all -- --check                                                           # clean

Notes

• Fixes an unbalanced brace that made deployment-examples/docker-compose/local-storage-cas-zstd.json5 unparseable, and extends nativelink-config/tests/json5_test.rs to parse every deployment example so a broken one can't ship again. (That test runs under native-cargo; it self-skips under bazel, where the examples aren't in runfiles.)
• nativelink-config/examples/stores-config.json5 is generated from the fenced json examples in the stores.rs doc comments by generate-stores-config, so the new fields are documented there and the example follows from it.
• The store publishes its own metrics — wire_uploads/wire_upload_bytes/wire_downloads, batch_zstd_passthroughs/batch_identity_decodes, inline_commits/staged_commits, the staged_uploads_inflight/identity_ops_inflight gauges, recompressions_applied/rejected/skipped_busy, and stage_timeouts/commit_timeouts — so the bounds above can be tuned from measurements rather than guesses.
• Follow-up worth doing separately: the identity encode/decode paths still run on spawn_blocking threads, which is what max_concurrent_identity_ops bounds. Add opt-in zstd wire compression to GrpcStore transfers #2596 converted the analogous service codecs to drive the zstd raw decoder from an async loop; adopting that here would remove the blocking-thread-per-transfer cost and let the knob go away.

🤖 Generated with Claude Code

---

This change is

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview	Jul 30, 2026 10:53pm
nativelink-aidm	Ready	Preview	Jul 30, 2026 10:53pm