Configure zstd passthrough through the compression store - #2
Draft
walter-zeromatter wants to merge 84 commits into
Draft
Configure zstd passthrough through the compression store#2walter-zeromatter wants to merge 84 commits into
walter-zeromatter wants to merge 84 commits into
Conversation
…a#2553) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
When `experimental_read_batching` is configured on a gRPC store, full reads of small blobs are coalesced into BatchReadBlobs RPCs instead of issuing one ByteStream Read stream per blob. Each ByteStream read carries a ~1.6ms fixed per-RPC cost; batched reads measured 52us/blob at 4KiB (30.2x), 4.8x at 32KiB and 1.8x at 256KiB against a real in-process gRPC server. The coalescer uses slot-based group commit with no timers: callers enqueue their read and try_acquire one of `dispatch_slots` semaphore permits to become a dispatcher. A dispatcher drains up to `max_batch_bytes` of pending reads (grouped by digest function, with a per-entry overhead charge to bound entry counts) into a single BatchReadBlobs request, deduplicates digests and fans response data out to every waiter of a digest, validates entry size and identity compression, and keeps draining until the queue is empty. After releasing its permit the dispatcher re-checks the queue to close the race with concurrent enqueues. When more than `max_queued_bytes` are already waiting, new reads bypass batching and use the stream path so the coalescer never blocks. Per-item failures (e.g. NOT_FOUND) fail only that read; retryable per-item errors fall back to the ByteStream path, which re-enters the retry machinery; whole-RPC failures are broadcast to the batch. Dispatchers run as detached background tasks holding a strong reference obtained via a weak self pointer: cancellation of any individual reader (timeouts, try_join siblings) can neither abort an in-flight batch RPC nor strand still-queued waiters - a cancelled caller only drops its own result receiver. Because batched reads share one upstream RPC across many client requests, `experimental_read_batching` is rejected at construction when combined with `forward_headers`, whose per-client values (e.g. credentials) cannot be attached to a shared RPC. The coalescer exports metrics under the store's read_batcher group: batches_sent, blobs_batched, queue_bypasses, batched_read_errors and a queued_bytes gauge (atomic mirror of the queue byte budget). Partial reads, blobs above `max_blob_size_bytes`, non-digest keys, AC stores and empty blobs keep the existing behavior. The feature is off by default and unset config is zero behavior change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Remove quoting around NetApp * Fix clippy permitted list of idents
Co-authored-by: config reference bot <bot@tracemachina.com>
…eader adapter (TraceMachina#2574) * test(service): compressed-download encode streams must not starve the blocking pool A compressed-download encode stream drains at the gRPC client's pace. The current implementation holds one tokio blocking-pool thread per stream for the stream's entire lifetime (BufChannelReader blocks waiting for input and blocking_send parks until the client consumes), so N concurrent downloads with slow consumers occupy N pool threads and every unrelated spawn_blocking user queues behind them. This test wires encode streams the way ByteStreamServer::inner_read_compressed does, holds more of them than the pool has threads, and asserts a trivial unrelated spawn_blocking closure still runs. It fails against the current thread-per-stream implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(service): encode compressed downloads asynchronously, off the blocking pool stream_encode_compressed_download previously ran under spawn_blocking with a blocking reader over raw_rx and blocking_send into the output channel, so one tokio blocking-pool thread was parked per compressed download for the whole stream at the client's drain rate. Concurrent downloads with slow consumers could exhaust the pool (default cap 512), starving every other spawn_blocking user — filesystem store I/O, upload decode, credential resolution — and the per-thread zstd contexts, stacks, and buffers inflated memory accordingly. The encoder is now an async fn driving zstd's raw streaming API (ZSTD_compressStream via zstd::stream::raw::Encoder): async recv, inline per-chunk encode (CPU bounded by the small channel chunk size), async send for backpressure. No thread is held while waiting on either channel. The wire format is unchanged: one well-formed zstd frame, identical to what zstd::stream::read::Encoder emitted. The bytestream_server caller drops its spawn_blocking wrapper; dropping the read stream still cancels the encode because the future itself is dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(service): decode compressed uploads asynchronously, off the blocking pool Same defect shape as the download encode path: stream_decode_compressed_upload ran under spawn_blocking, blocking on compressed_rx for client bytes and parking in blocking_send while the store drained, holding one blocking-pool thread per compressed upload for the stream's lifetime. The decoder is now an async fn driving zstd's raw streaming API (ZSTD_decompressStream via zstd::stream::raw::Decoder) with async channel sends. Validation is unchanged: the per-chunk decoded-size cap (bomb rejection), the exact final-size check, and the digest check are preserved, and a stream that ends mid-frame is rejected as InvalidArgument (previously surfaced as a read error from the blocking decoder). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(service): remove unused BufChannelReader adapter The async streaming zstd rewrite left this blocking Read adapter with no production users; drop it and its unit test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Walter Gray <walter@0m.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a#2576) Co-authored-by: config reference bot <bot@tracemachina.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
* Work towards dealing with redis eviction * Boot redis subscription manager on start * Handle eviction events * Refactor RemoveItemCallback type * Run remove callbacks for Redis * Add redis_store_tester support for Redis eviction events * Fix some build issues * Add redis store eviction tests * Add docs about required new redis config * Better config get parsing * Fix redis key decoding * Test with redis_store plus existence cache --------- Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
A read that failed with NotFound returned RetryResult::Err, which bypassed the retrier entirely. This made a read-404 terminal even though the store's retry config can classify NotFound as retryable via retry_on_errors, so read-after-write races (an object still finalizing or being repopulated after eviction) could never be retried at the store level and had to be absorbed by scheduler re-dispatch, exhausting max_job_retries under concurrent load. Emit RetryResult::Retry instead and let the retrier classify the error. Default behavior is unchanged: the retrier does not retry NotFound unless retry_on_errors opts in, and has() still maps NotFound to Ok(None) for existence probes. Fixes TraceMachina#2582
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
) The FAQ answered conceptual questions (cost, caching, LRE, Rust) but nothing operational. Add eight question-phrased pages distilled from the rest of the docs — client wiring, configuration, store selection, production deployment, observability, troubleshooting, architecture, and contributing — each grounded in and linking back to the full page it summarizes. Also add a "Contributing to Docs or the Website" section to CONTRIBUTING.md: the old web/platform bun setup/docs/preview workflow no longer exists, and nothing documented the current Bun + Turborepo web/ workspace or that it sits outside the Bazel build. Signed-off-by: Alec Maliwanag <amaliwanag@gmail.com>
Third-party tags load through the existing Google Tag Manager container instead of inline scripts.
…#2637) Removes the Cloud pricing tier, comparison column, dev.nativelink.com signup links, and remaining NativeLink Cloud mentions from the marketing site and docs. Enterprise remains the managed option; internal CI usage and historical changelogs are untouched. Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
…#2596) * Add opt-in wire compression to GrpcStore's own transfers NativeLink's internal hops never used REAPI compressed-blobs: GrpcStore built plain blobs/ resource names for every upload and read, so worker output uploads and input fetches shipped raw bytes in both directions regardless of any compression setting (the zstd support from the server work is accept/advertise only). New opt-in GrpcSpec.experimental_remote_cache_compression: uploads and full-blob downloads of blobs at or above 64KiB stream through zstd (level 1; higher levels measured slower with no meaningful wire reduction) using compressed-blobs/zstd resource names. Ranged reads and small blobs keep the identity path. Downloads decode with size and digest verification; a retryable mid-stream failure resumes via the identity path at the already-forwarded uncompressed offset. The shared streaming codecs move from nativelink-service to nativelink-util (the service re-exports them), since nativelink-store cannot depend on the service crate. Also fixes a latent decoder bug found by the new tests: a blob whose decompressed size is an exact multiple of the 128KiB decoder output buffer made the loop poll the finished decoder once more, and the new-frame input hint made the EOF check misreport a fully decoded stream as "ended in the middle of a zstd frame" — spuriously failing compressed uploads of exact-multiple-size blobs from any client. Measured (token-bucket throttled real TCP, byte-verified, off -> on): 96MiB 3.7:1-compressible blob @1gbps: upload 712ms -> 203ms and download 706ms -> 204ms (3.5x), wire 101MB -> 27MB; @100Mbps 24MiB: 1975ms -> 494ms (4.0x). Incompressible payloads: identical wall time, no size penalty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC * Make wire compression streaming and cancellation safe * Fix clippy items-after-statements and buildifier ordering Hoist the two function-local enums above the first statements of their functions and restore alphabetical order in the service test suite list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC * Fix clippy cast_possible_wrap in compressed upload accounting Match the identity path's idiom from the cast_possible_wrap cleanup on main: saturating try_into instead of a bare usize-to-i64 cast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
…semaphore permits (TraceMachina#2636) (TraceMachina#2638) Signed-off-by: namdamdoi68-oss <namdamdoi68@gmail.com> Co-authored-by: Tom Parker-Shemilt <tom@tracemachina.com>
Adds a downcast helper that checks only the immediate inner StoreDriver, without following inner_store() like the existing recursive downcast_ref. Needed so callers (e.g. the service layer) can detect a directly-configured representation-changing store such as the upcoming ZstdStore at an instance boundary, rather than resolving through pass-through wrappers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nativelink-store failed to compile (E0004) because the StoreSpec::ZstdStore variant had no match arm in default_store_factory.rs. Add the arm, wiring ZstdStore::new to its backend store like the other pass-through stores, and add a factory test that builds a ZstdStore via store_factory and downcasts to confirm the concrete type.
…dation tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… concat-frame test Register staged/recompression temp file paths in TempFileGuard before create_empty_temp_file runs, so a set_permissions failure or cancellation during file creation can't leak an untracked file. Add a post_init write-probe so a read-only or mispermissioned temp_path fails at startup instead of on first upload. Document that the 0o600 permission is unix-only. Add a test proving get_for_batch's raw-decode path handles a concatenated multi-frame zstd physical stream.
…rite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sion The ZstdStore fast path in inner_batch_update_blobs took a client's zstd-compressed blob and stored it verbatim via update_zstd_oneshot whenever the instance's CAS store happened to be a ZstdStore, without checking remote_cache_compression_enabled. This let a ZstdStore-backed instance silently accept zstd uploads even when remote cache compression was disabled for it, unlike the non-ZstdStore path (which already rejects zstd via decompress_batch_update) and capabilities (which stops advertising zstd when disabled). Require remote_cache_compression_enabled alongside the ZstdStore/Zstd compressor check so a disabled instance falls through to decompress_batch_update and gets the same rejection as any other store. Also moved the size_bytes computation into the else branch since it is only used there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fast-path sites Add a non-recursive owned-Arc sibling to downcast_ref_immediate for callers that must move the concrete store into a spawned/'static future, and replace the four repeated `store.clone().into_inner().as_any_arc().downcast::<ZstdStore>()` idioms in the ByteStream and CAS servers with it. Behavior is unchanged: into_inner() returns the immediate inner Arc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ZstdStore's identity encode/decode paths run inside `spawn_blocking!` and need `std::io::Read`/`Write` views over a buf_channel pair. Add `BufChannelReader`/`BufChannelWriter` to `nativelink_util::buf_channel`, where the underlying `DropCloserReadHalf`/`DropCloserWriteHalf` already live, and drop the private copies from zstd_store.rs. `BufChannelWriter` deliberately does not send EOF on drop: the caller sends it explicitly once an upload has been validated, so a failed write never commits downstream. The service wire codecs used to need the same adapter, but no longer do — they drive the zstd raw decoder from an async loop instead of a blocking one — so these adapters now exist solely for the store's blocking paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tore Introduce a single private decode_all_zstd(data, size_hint, on_err) helper and use it at the three whole-buffer decode sites (two zstd::stream::decode_all and one zstd::bulk::decompress). The size_hint selects bulk vs streaming decode and the on_err closure preserves each site's exact error code and message (InvalidArgument for client input, DataLoss for stored data). The two streaming decoders are left untouched. Behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under Bazel (unlike cargo, which inherits crate deps) the store integration test suite needs `@crates//:zstd` declared explicitly for the ZstdStore tests. Caught by the clippy/build aspect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pinned staging Harden the ZstdStore fast path against resource-exhaustion, starvation, and staging path-replacement races, and clean the patch-specific CI failures. Issue 1 — bounded decompression at the output sink: - DecodeSink enforces the decoded-output ceiling inside Write::write using checked_add; output past the digest's uncompressed size is rejected immediately (InvalidArgument) before hashing/collecting, stopping a zstd bomb at the first over-limit block. - Zero-digest validation streams through the same bounded decoder with an output cap of zero (no whole-buffer decode/allocation) and now acquires the staged-upload semaphore, so it participates in concurrency admission. Issue 2 — slow-client / stall starvation: - The compressed ByteStream client pump applies an idle timeout to waiting for the next WriteRequest (reusing persist_stream_on_disconnect_timeout_s); a client making progress is never timed out. On timeout the channel sender is dropped so the blocking validation unwinds through the existing join, and DeadlineExceeded is surfaced as the primary error. - Inner-store commit (and recompression) is bounded by a new commit_timeout_s (default 300s); on expiry the upload fails DeadlineExceeded and the staged file/permit are released, so a stalled backend cannot hold a slot forever. Issue 3 — descriptor-pinned staging: - Staging files are created exclusively (create_new/O_EXCL, 0o600 atomic on unix); the validated descriptor is retained, rewound, and handed to the inner store at commit (never reopened by pathname), defeating observe-and-replace races. Recompression candidate handled the same way. - post_init verifies temp_path is a directory and rejects a world-writable directory lacking the sticky bit. Adds fs::FileSlot::from_std. Issue 4/5 — CI + docs: - Regenerate stores-config.json5, add Vale vocabulary terms, fix MDX-unsafe comparison operators (windowLog <= 23 -> ≤), and reconcile store-overview docs with the new bounds/admission/deadline/temp_path/descriptor behavior. Adds focused tests: sink bound + zero-digest bomb + over-decode rejection, descriptor-pinned commit, commit-timeout slot release (store); ByteStream idle-timeout slot release + progress-not-timed-out (service); batch per-blob timeout sibling isolation (service). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correctness and availability: - Qualify the descriptor-pinning claim. Only backends using the default `update_with_whole_file` read the validated descriptor; `filesystem` drops it and commits by `rename(2)` on the path, where the defence is `O_EXCL` creation of an unguessable name in an operator-private dir. Adds a filesystem-backed commit test, which the previous `MemoryStore`-only test could not cover. - Report the staging error, not the feeder's consequential channel error, when a large oneshot upload is rejected. - Route the negotiated compressor through a single `wire_compressor_capability` helper instead of hardcoding zstd in the ByteStream read/write and BatchUpdate fast paths. The helper lives in `nativelink_util::wire_compression` next to the codecs and `WireCompressor`, and is re-exported from the service module. - Add `stage_timeout_s`: a total validate-and-stage deadline, since a per-message idle timeout is reset by a client that trickles bytes to hold a staging slot open. - Make recompression best-effort (`try_acquire`). It previously queued on the recompression semaphore while holding a staging permit, letting a pool of 1 throttle the whole upload path. - Reject `max_recompression_size > 0` without `compression_level`, which silently disabled recompression. - Add `max_concurrent_identity_ops` to bound identity reads/writes, each of which holds a blocking thread for a whole transfer. - Add `max_inline_commit_size`: validate and commit small compressed uploads from memory, so BatchUpdateBlobs stops paying a per-blob fsync. - Give compressed uploads their own `compressed_upload_idle_timeout_s` instead of borrowing `persist_stream_on_disconnect_timeout_s`. - Publish metrics for fast-path hits, inline vs staged commits, in-flight gauges, recompression outcomes, and deadline expirations. Fix an unbalanced brace that made `deployment-examples/docker-compose/local-storage-cas-zstd.json5` unparseable, and extend the json5 test to parse every deployment example so nothing ships broken again. Drop stale references to the abandoned two-candidate staging design, consolidate the duplicated join/error/decode helpers, and cut the comment density from 18% to 15% while adding the above. Also adapts the branch to upstream API changes picked up by the rebase: the `RemoveCallback` alias, `FilesystemSpec::evict_page_cache`, and the newly denied `clippy::cast_possible_wrap` in tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
walter-zeromatter
force-pushed
the
user/wgray/zstd-store
branch
from
July 30, 2026 21:41
e5b3e0d to
22bf38b
Compare
Three CI failures, all in files this branch touches: - rustfmt (the pinned nightly used by the Bazel aspect) reflows the `store_trait` import and the `register_remove_callback` signature in zstd_store.rs, both of which changed width when the rebase adopted the shorter `RemoveCallback` alias. It also wants the new `use crate::store_trait::WireCompressor` sorted after `digest_hasher` rather than after `buf_channel`. - `nativelink-config/examples/stores-config.json5` is generated from the ```json blocks in stores.rs doc comments by `generate-stores-config`; it is not hand-editable. Add `max_concurrent_identity_ops` and `max_inline_commit_size` to the doc comment, which is the source of truth, so the generator reproduces the committed file exactly. - Vale lints Rust doc comments as well as MDX. Rephrase to avoid the possessives `backend's`/`upload's` and the two words absent from its dictionary (`untrusted`, `expirations`) instead of widening the accepted vocabulary for ordinary prose. Verified with `bazel test //...` (103/103, so the `unit_test` targets that surfaced the rustfmt failures are covered this time) plus the cargo test, clippy, and nightly rustfmt runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
compression_algorithm.zstdon the existingcompressionstore configuration instead of introducing a new public store type.--remote_cache_compressionclients the stored stream byte-for-byte.Why
Compression is already modeled as a store with an algorithm choice, so zstd belongs on that existing configuration surface alongside LZ4. The zstd implementation remains specialized internally because it can reuse the stored frame on the REAPI wire; that behavior is exposed as a store capability rather than as a service dependency on a new concrete store type.
The capability is intentionally immediate. Configure the compression store as the instance CAS boundary to enable byte-for-byte passthrough. An outer wrapper remains correct but uses the normal decode/re-encode fallback at that boundary.
Configuration
Use a new or empty dedicated backend namespace for this representation. Rollout and rollback require clearing that cache namespace; there is no in-place migration from identity or LZ4 storage.
Validation
bazel test //nativelink-config:unit_test //nativelink-config:integration_tests/json5_test_test //nativelink-util:integration_tests/store_trait_test_test //nativelink-store:integration_tests/zstd_store_test_test //nativelink-service:integration_tests/bytestream_server_test_test //nativelink-service:integration_tests/cas_server_test_testcargo +nightly fmt --all -- --checkgit diff --checkThe focused Bazel suite passes all six targets, including their Clippy aspects. Vale was not available in the local environment, so documentation lint was not run locally.