Skip to content

Configure zstd passthrough through the compression store - #2

Draft
walter-zeromatter wants to merge 84 commits into
mainfrom
user/wgray/zstd-store
Draft

Configure zstd passthrough through the compression store#2
walter-zeromatter wants to merge 84 commits into
mainfrom
user/wgray/zstd-store

Conversation

@walter-zeromatter

@walter-zeromatter walter-zeromatter commented Jul 15, 2026

Copy link
Copy Markdown

Summary

  • Add zstd as compression_algorithm.zstd on the existing compression store configuration instead of introducing a new public store type.
  • Keep blobs as standard zstd streams at rest and serve eligible --remote_cache_compression clients the stored stream byte-for-byte.
  • Expose zstd wire operations through an immediate store capability, so ByteStream and CAS services do not import or downcast to the concrete implementation.
  • Preserve bounded staging, upload timeouts, digest validation, recompression limits, and identity-client fallback behavior.
  • Update configuration examples and operator documentation for the unified compression-store shape.

Why

Compression is already modeled as a store with an algorithm choice, so zstd belongs on that existing configuration surface alongside LZ4. The zstd implementation remains specialized internally because it can reuse the stored frame on the REAPI wire; that behavior is exposed as a store capability rather than as a service dependency on a new concrete store type.

The capability is intentionally immediate. Configure the compression store as the instance CAS boundary to enable byte-for-byte passthrough. An outer wrapper remains correct but uses the normal decode/re-encode fallback at that boundary.

Configuration

{
  compression: {
    compression_algorithm: {
      zstd: {
        temp_path: "/tmp/nativelink-zstd",
        max_compressed_upload_size: "512MiB",
      },
    },
    backend: {
      memory: {},
    },
  },
}

Use a new or empty dedicated backend namespace for this representation. Rollout and rollback require clearing that cache namespace; there is no in-place migration from identity or LZ4 storage.

Validation

  • bazel test //nativelink-config:unit_test //nativelink-config:integration_tests/json5_test_test //nativelink-util:integration_tests/store_trait_test_test //nativelink-store:integration_tests/zstd_store_test_test //nativelink-service:integration_tests/bytestream_server_test_test //nativelink-service:integration_tests/cas_server_test_test
  • cargo +nightly fmt --all -- --check
  • git diff --check

The focused Bazel suite passes all six targets, including their Clippy aspects. Vale was not available in the local environment, so documentation lint was not run locally.

palfrey and others added 30 commits July 15, 2026 18:14
…a#2553)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
When `experimental_read_batching` is configured on a gRPC store, full
reads of small blobs are coalesced into BatchReadBlobs RPCs instead of
issuing one ByteStream Read stream per blob. Each ByteStream read
carries a ~1.6ms fixed per-RPC cost; batched reads measured 52us/blob
at 4KiB (30.2x), 4.8x at 32KiB and 1.8x at 256KiB against a real
in-process gRPC server.

The coalescer uses slot-based group commit with no timers: callers
enqueue their read and try_acquire one of `dispatch_slots` semaphore
permits to become a dispatcher. A dispatcher drains up to
`max_batch_bytes` of pending reads (grouped by digest function, with a
per-entry overhead charge to bound entry counts) into a single
BatchReadBlobs request, deduplicates digests and fans response data out
to every waiter of a digest, validates entry size and identity
compression, and keeps draining until the queue is empty. After
releasing its permit the dispatcher re-checks the queue to close the
race with concurrent enqueues. When more than `max_queued_bytes` are
already waiting, new reads bypass batching and use the stream path so
the coalescer never blocks. Per-item failures (e.g. NOT_FOUND) fail
only that read; retryable per-item errors fall back to the ByteStream
path, which re-enters the retry machinery; whole-RPC failures are
broadcast to the batch.

Dispatchers run as detached background tasks holding a strong reference
obtained via a weak self pointer: cancellation of any individual reader
(timeouts, try_join siblings) can neither abort an in-flight batch RPC
nor strand still-queued waiters - a cancelled caller only drops its own
result receiver. Because batched reads share one upstream RPC across
many client requests, `experimental_read_batching` is rejected at
construction when combined with `forward_headers`, whose per-client
values (e.g. credentials) cannot be attached to a shared RPC.

The coalescer exports metrics under the store's read_batcher group:
batches_sent, blobs_batched, queue_bypasses, batched_read_errors and a
queued_bytes gauge (atomic mirror of the queue byte budget).

Partial reads, blobs above `max_blob_size_bytes`, non-digest keys, AC
stores and empty blobs keep the existing behavior. The feature is off
by default and unset config is zero behavior change.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Remove quoting around NetApp
* Fix clippy permitted list of idents
Co-authored-by: config reference bot <bot@tracemachina.com>
…eader adapter (TraceMachina#2574)

* test(service): compressed-download encode streams must not starve the blocking pool

A compressed-download encode stream drains at the gRPC client's pace. The
current implementation holds one tokio blocking-pool thread per stream for
the stream's entire lifetime (BufChannelReader blocks waiting for input and
blocking_send parks until the client consumes), so N concurrent downloads
with slow consumers occupy N pool threads and every unrelated spawn_blocking
user queues behind them. This test wires encode streams the way
ByteStreamServer::inner_read_compressed does, holds more of them than the
pool has threads, and asserts a trivial unrelated spawn_blocking closure
still runs. It fails against the current thread-per-stream implementation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(service): encode compressed downloads asynchronously, off the blocking pool

stream_encode_compressed_download previously ran under spawn_blocking with a
blocking reader over raw_rx and blocking_send into the output channel, so one
tokio blocking-pool thread was parked per compressed download for the whole
stream at the client's drain rate. Concurrent downloads with slow consumers
could exhaust the pool (default cap 512), starving every other spawn_blocking
user — filesystem store I/O, upload decode, credential resolution — and the
per-thread zstd contexts, stacks, and buffers inflated memory accordingly.

The encoder is now an async fn driving zstd's raw streaming API
(ZSTD_compressStream via zstd::stream::raw::Encoder): async recv, inline
per-chunk encode (CPU bounded by the small channel chunk size), async send
for backpressure. No thread is held while waiting on either channel. The
wire format is unchanged: one well-formed zstd frame, identical to what
zstd::stream::read::Encoder emitted. The bytestream_server caller drops its
spawn_blocking wrapper; dropping the read stream still cancels the encode
because the future itself is dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(service): decode compressed uploads asynchronously, off the blocking pool

Same defect shape as the download encode path: stream_decode_compressed_upload
ran under spawn_blocking, blocking on compressed_rx for client bytes and
parking in blocking_send while the store drained, holding one blocking-pool
thread per compressed upload for the stream's lifetime.

The decoder is now an async fn driving zstd's raw streaming API
(ZSTD_decompressStream via zstd::stream::raw::Decoder) with async channel
sends. Validation is unchanged: the per-chunk decoded-size cap (bomb
rejection), the exact final-size check, and the digest check are preserved,
and a stream that ends mid-frame is rejected as InvalidArgument (previously
surfaced as a read error from the blocking decoder).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(service): remove unused BufChannelReader adapter

The async streaming zstd rewrite left this blocking Read adapter with no
production users; drop it and its unit test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Walter Gray <walter@0m.dev>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a#2576)

Co-authored-by: config reference bot <bot@tracemachina.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
* Work towards dealing with redis eviction
* Boot redis subscription manager on start
* Handle eviction events
* Refactor RemoveItemCallback type
* Run remove callbacks for Redis
* Add redis_store_tester support for Redis eviction events
* Fix some build issues
* Add redis store eviction tests
* Add docs about required new redis config
* Better config get parsing
* Fix redis key decoding

* Test with redis_store plus existence cache

---------

Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
A read that failed with NotFound returned RetryResult::Err, which
bypassed the retrier entirely. This made a read-404 terminal even
though the store's retry config can classify NotFound as retryable
via retry_on_errors, so read-after-write races (an object still
finalizing or being repopulated after eviction) could never be
retried at the store level and had to be absorbed by scheduler
re-dispatch, exhausting max_job_retries under concurrent load.

Emit RetryResult::Retry instead and let the retrier classify the
error. Default behavior is unchanged: the retrier does not retry
NotFound unless retry_on_errors opts in, and has() still maps
NotFound to Ok(None) for existence probes.

Fixes TraceMachina#2582
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
)

The FAQ answered conceptual questions (cost, caching, LRE, Rust) but
nothing operational. Add eight question-phrased pages distilled from
the rest of the docs — client wiring, configuration, store selection,
production deployment, observability, troubleshooting, architecture,
and contributing — each grounded in and linking back to the full page
it summarizes.

Also add a "Contributing to Docs or the Website" section to
CONTRIBUTING.md: the old web/platform bun setup/docs/preview workflow
no longer exists, and nothing documented the current Bun + Turborepo
web/ workspace or that it sits outside the Bazel build.

Signed-off-by: Alec Maliwanag <amaliwanag@gmail.com>
MarcusSorealheis and others added 28 commits July 27, 2026 11:17
Third-party tags load through the existing Google Tag Manager container
instead of inline scripts.
…#2637)

Removes the Cloud pricing tier, comparison column, dev.nativelink.com
signup links, and remaining NativeLink Cloud mentions from the
marketing site and docs. Enterprise remains the managed option;
internal CI usage and historical changelogs are untouched.

Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
…#2596)

* Add opt-in wire compression to GrpcStore's own transfers

NativeLink's internal hops never used REAPI compressed-blobs: GrpcStore
built plain blobs/ resource names for every upload and read, so worker
output uploads and input fetches shipped raw bytes in both directions
regardless of any compression setting (the zstd support from the server
work is accept/advertise only).

New opt-in GrpcSpec.experimental_remote_cache_compression: uploads and
full-blob downloads of blobs at or above 64KiB stream through zstd
(level 1; higher levels measured slower with no meaningful wire
reduction) using compressed-blobs/zstd resource names. Ranged reads and
small blobs keep the identity path. Downloads decode with size and
digest verification; a retryable mid-stream failure resumes via the
identity path at the already-forwarded uncompressed offset. The shared
streaming codecs move from nativelink-service to nativelink-util (the
service re-exports them), since nativelink-store cannot depend on the
service crate.

Also fixes a latent decoder bug found by the new tests: a blob whose
decompressed size is an exact multiple of the 128KiB decoder output
buffer made the loop poll the finished decoder once more, and the
new-frame input hint made the EOF check misreport a fully decoded
stream as "ended in the middle of a zstd frame" — spuriously failing
compressed uploads of exact-multiple-size blobs from any client.

Measured (token-bucket throttled real TCP, byte-verified, off -> on):
96MiB 3.7:1-compressible blob @1gbps: upload 712ms -> 203ms and
download 706ms -> 204ms (3.5x), wire 101MB -> 27MB; @100Mbps 24MiB:
1975ms -> 494ms (4.0x). Incompressible payloads: identical wall time,
no size penalty.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC

* Make wire compression streaming and cancellation safe

* Fix clippy items-after-statements and buildifier ordering

Hoist the two function-local enums above the first statements of their
functions and restore alphabetical order in the service test suite list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC

* Fix clippy cast_possible_wrap in compressed upload accounting

Match the identity path's idiom from the cast_possible_wrap cleanup on
main: saturating try_into instead of a bare usize-to-i64 cast.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
…semaphore permits (TraceMachina#2636) (TraceMachina#2638)

Signed-off-by: namdamdoi68-oss <namdamdoi68@gmail.com>
Co-authored-by: Tom Parker-Shemilt <tom@tracemachina.com>
Adds a downcast helper that checks only the immediate inner StoreDriver,
without following inner_store() like the existing recursive downcast_ref.
Needed so callers (e.g. the service layer) can detect a directly-configured
representation-changing store such as the upcoming ZstdStore at an instance
boundary, rather than resolving through pass-through wrappers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nativelink-store failed to compile (E0004) because the StoreSpec::ZstdStore
variant had no match arm in default_store_factory.rs. Add the arm, wiring
ZstdStore::new to its backend store like the other pass-through stores, and
add a factory test that builds a ZstdStore via store_factory and downcasts
to confirm the concrete type.
…dation tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… concat-frame test

Register staged/recompression temp file paths in TempFileGuard before
create_empty_temp_file runs, so a set_permissions failure or cancellation
during file creation can't leak an untracked file. Add a post_init
write-probe so a read-only or mispermissioned temp_path fails at startup
instead of on first upload. Document that the 0o600 permission is unix-only.
Add a test proving get_for_batch's raw-decode path handles a concatenated
multi-frame zstd physical stream.
…rite

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sion

The ZstdStore fast path in inner_batch_update_blobs took a client's
zstd-compressed blob and stored it verbatim via update_zstd_oneshot
whenever the instance's CAS store happened to be a ZstdStore, without
checking remote_cache_compression_enabled. This let a ZstdStore-backed
instance silently accept zstd uploads even when remote cache
compression was disabled for it, unlike the non-ZstdStore path (which
already rejects zstd via decompress_batch_update) and capabilities
(which stops advertising zstd when disabled).

Require remote_cache_compression_enabled alongside the ZstdStore/Zstd
compressor check so a disabled instance falls through to
decompress_batch_update and gets the same rejection as any other
store. Also moved the size_bytes computation into the else branch
since it is only used there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fast-path sites

Add a non-recursive owned-Arc sibling to downcast_ref_immediate for callers
that must move the concrete store into a spawned/'static future, and replace
the four repeated `store.clone().into_inner().as_any_arc().downcast::<ZstdStore>()`
idioms in the ByteStream and CAS servers with it. Behavior is unchanged:
into_inner() returns the immediate inner Arc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ZstdStore's identity encode/decode paths run inside `spawn_blocking!` and
need `std::io::Read`/`Write` views over a buf_channel pair. Add
`BufChannelReader`/`BufChannelWriter` to `nativelink_util::buf_channel`,
where the underlying `DropCloserReadHalf`/`DropCloserWriteHalf` already
live, and drop the private copies from zstd_store.rs.

`BufChannelWriter` deliberately does not send EOF on drop: the caller sends
it explicitly once an upload has been validated, so a failed write never
commits downstream.

The service wire codecs used to need the same adapter, but no longer do —
they drive the zstd raw decoder from an async loop instead of a blocking
one — so these adapters now exist solely for the store's blocking paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tore

Introduce a single private decode_all_zstd(data, size_hint, on_err) helper and
use it at the three whole-buffer decode sites (two zstd::stream::decode_all and
one zstd::bulk::decompress). The size_hint selects bulk vs streaming decode and
the on_err closure preserves each site's exact error code and message
(InvalidArgument for client input, DataLoss for stored data). The two streaming
decoders are left untouched. Behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under Bazel (unlike cargo, which inherits crate deps) the store
integration test suite needs `@crates//:zstd` declared explicitly for the
ZstdStore tests. Caught by the clippy/build aspect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pinned staging

Harden the ZstdStore fast path against resource-exhaustion, starvation, and
staging path-replacement races, and clean the patch-specific CI failures.

Issue 1 — bounded decompression at the output sink:
- DecodeSink enforces the decoded-output ceiling inside Write::write using
  checked_add; output past the digest's uncompressed size is rejected
  immediately (InvalidArgument) before hashing/collecting, stopping a zstd
  bomb at the first over-limit block.
- Zero-digest validation streams through the same bounded decoder with an
  output cap of zero (no whole-buffer decode/allocation) and now acquires the
  staged-upload semaphore, so it participates in concurrency admission.

Issue 2 — slow-client / stall starvation:
- The compressed ByteStream client pump applies an idle timeout to waiting for
  the next WriteRequest (reusing persist_stream_on_disconnect_timeout_s); a
  client making progress is never timed out. On timeout the channel sender is
  dropped so the blocking validation unwinds through the existing join, and
  DeadlineExceeded is surfaced as the primary error.
- Inner-store commit (and recompression) is bounded by a new commit_timeout_s
  (default 300s); on expiry the upload fails DeadlineExceeded and the staged
  file/permit are released, so a stalled backend cannot hold a slot forever.

Issue 3 — descriptor-pinned staging:
- Staging files are created exclusively (create_new/O_EXCL, 0o600 atomic on
  unix); the validated descriptor is retained, rewound, and handed to the
  inner store at commit (never reopened by pathname), defeating
  observe-and-replace races. Recompression candidate handled the same way.
- post_init verifies temp_path is a directory and rejects a world-writable
  directory lacking the sticky bit. Adds fs::FileSlot::from_std.

Issue 4/5 — CI + docs:
- Regenerate stores-config.json5, add Vale vocabulary terms, fix MDX-unsafe
  comparison operators (windowLog <= 23 -> ≤), and reconcile store-overview
  docs with the new bounds/admission/deadline/temp_path/descriptor behavior.

Adds focused tests: sink bound + zero-digest bomb + over-decode rejection,
descriptor-pinned commit, commit-timeout slot release (store); ByteStream
idle-timeout slot release + progress-not-timed-out (service); batch per-blob
timeout sibling isolation (service).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correctness and availability:

- Qualify the descriptor-pinning claim. Only backends using the default
  `update_with_whole_file` read the validated descriptor; `filesystem`
  drops it and commits by `rename(2)` on the path, where the defence is
  `O_EXCL` creation of an unguessable name in an operator-private dir.
  Adds a filesystem-backed commit test, which the previous
  `MemoryStore`-only test could not cover.
- Report the staging error, not the feeder's consequential channel error,
  when a large oneshot upload is rejected.
- Route the negotiated compressor through a single
  `wire_compressor_capability` helper instead of hardcoding zstd in the
  ByteStream read/write and BatchUpdate fast paths. The helper lives in
  `nativelink_util::wire_compression` next to the codecs and
  `WireCompressor`, and is re-exported from the service module.
- Add `stage_timeout_s`: a total validate-and-stage deadline, since a
  per-message idle timeout is reset by a client that trickles bytes to
  hold a staging slot open.
- Make recompression best-effort (`try_acquire`). It previously queued on
  the recompression semaphore while holding a staging permit, letting a
  pool of 1 throttle the whole upload path.
- Reject `max_recompression_size > 0` without `compression_level`, which
  silently disabled recompression.
- Add `max_concurrent_identity_ops` to bound identity reads/writes, each
  of which holds a blocking thread for a whole transfer.
- Add `max_inline_commit_size`: validate and commit small compressed
  uploads from memory, so BatchUpdateBlobs stops paying a per-blob fsync.
- Give compressed uploads their own `compressed_upload_idle_timeout_s`
  instead of borrowing `persist_stream_on_disconnect_timeout_s`.
- Publish metrics for fast-path hits, inline vs staged commits, in-flight
  gauges, recompression outcomes, and deadline expirations.

Fix an unbalanced brace that made
`deployment-examples/docker-compose/local-storage-cas-zstd.json5`
unparseable, and extend the json5 test to parse every deployment example
so nothing ships broken again.

Drop stale references to the abandoned two-candidate staging design,
consolidate the duplicated join/error/decode helpers, and cut the comment
density from 18% to 15% while adding the above.

Also adapts the branch to upstream API changes picked up by the rebase:
the `RemoveCallback` alias, `FilesystemSpec::evict_page_cache`, and the
newly denied `clippy::cast_possible_wrap` in tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three CI failures, all in files this branch touches:

- rustfmt (the pinned nightly used by the Bazel aspect) reflows the
  `store_trait` import and the `register_remove_callback` signature in
  zstd_store.rs, both of which changed width when the rebase adopted the
  shorter `RemoveCallback` alias. It also wants the new
  `use crate::store_trait::WireCompressor` sorted after `digest_hasher`
  rather than after `buf_channel`.

- `nativelink-config/examples/stores-config.json5` is generated from the
  ```json blocks in stores.rs doc comments by `generate-stores-config`; it
  is not hand-editable. Add `max_concurrent_identity_ops` and
  `max_inline_commit_size` to the doc comment, which is the source of
  truth, so the generator reproduces the committed file exactly.

- Vale lints Rust doc comments as well as MDX. Rephrase to avoid the
  possessives `backend's`/`upload's` and the two words absent from its
  dictionary (`untrusted`, `expirations`) instead of widening the accepted
  vocabulary for ordinary prose.

Verified with `bazel test //...` (103/103, so the `unit_test` targets that
surfaced the rustfmt failures are covered this time) plus the cargo test,
clippy, and nightly rustfmt runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.