[9.1.1] Support bounded parallel chunk transfers (https://github.com/bazelbui… - #29614
Merged
Merged
Conversation
For `--experimental_remote_cache_chunking` implemented in bazelbuild#28437 This PR enables parallel uploads and downloads for chunked files to improve performance. Transport-level concurrency is still bounded by the existing remote cache gRPC connection/concurrency limits. The chunk transfer managers add a fixed per-blob window of 16 to prevent a single large blob from fanning out too aggressively. To avoid issues with batches, uploads and downloads use simple sliding-window style transfer managers. RELNOTES: CDC chunk uploads and downloads can now happen in parallel within a large blob. Benchmarks were rerun on 2026-05-12 with the JMH target `//src/test/java/com/google/devtools/build/lib/remote:ChunkedTransferBenchmark`. The parent baseline was measured by running the same benchmark harness against the parent commit. With the synthetic benchmark of network delays and simulated jitter, the current branch is about 11-13x faster than the parent baseline for these cases. As usual, this synthetic benchmark is not a substitute for real remote-cache measurements. After Change: ``` Benchmark (avgChunkSizeBytes) (chunkCount) (chunkSizeBytes) (delayMillis) (fileSizeBytes) (jitterMillis) (schedulerThreads) Mode Cnt Score Error Units ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 1 avgt 3 67.963 ± 2.039 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 2 avgt 3 67.925 ± 2.257 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 4 avgt 3 67.915 ± 2.497 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 8 avgt 3 67.946 ± 2.863 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 1 avgt 3 61.357 ± 7.385 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 2 avgt 3 61.373 ± 7.920 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 4 avgt 3 61.326 ± 8.220 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 8 avgt 3 61.393 ± 8.376 ms/op ``` Before Change: ``` Benchmark (avgChunkSizeBytes) (chunkCount) (chunkSizeBytes) (delayMillis) (fileSizeBytes) (jitterMillis) (schedulerThreads) Mode Cnt Score Error Units ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 1 avgt 3 812.081 ± 463.666 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 2 avgt 3 812.570 ± 442.021 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 4 avgt 3 811.883 ± 459.534 ms/op ChunkedTransferBenchmark.downloadChunked N/A 32 1024 25 N/A 10 8 avgt 3 812.371 ± 461.231 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 1 avgt 3 740.734 ± 389.653 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 2 avgt 3 742.434 ± 412.117 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 4 avgt 3 742.483 ± 395.466 ms/op ChunkedTransferBenchmark.uploadChunked 1024 N/A N/A 25 32768 10 8 avgt 3 742.509 ± 397.122 ms/op ``` Big File: ``` CURRENT BRANCH (512 MiB) Benchmark (avgChunkSizeBytes) (chunkCount) (chunkSizeBytes) (delayMillis) (fileSizeBytes) (jitterMillis) (schedulerThreads) Mode Cnt Score Error Units ChunkedTransferBenchmark.downloadChunked N/A 512 1048576 25 N/A 10 8 avgt 3 991.101 ± 173.890 ms/op ChunkedTransferBenchmark.uploadChunked 1048576 N/A N/A 25 536870912 10 8 avgt 3 1087.504 ± 123.857 ms/op ``` ``` PARENT BASELINE (512 MiB) Benchmark (avgChunkSizeBytes) (chunkCount) (chunkSizeBytes) (delayMillis) (fileSizeBytes) (jitterMillis) (schedulerThreads) Mode Cnt Score Error Units ChunkedTransferBenchmark.downloadChunked N/A 512 1048576 25 N/A 10 8 avgt 3 12849.136 ± 1733.063 ms/op ChunkedTransferBenchmark.uploadChunked 1048576 N/A N/A 25 536870912 10 8 avgt 3 11888.109 ± 2591.883 ms/op ``` Closes bazelbuild#29341. PiperOrigin-RevId: 919144974 Change-Id: Iaa0ca8971bd21c879f21c708327b5ddd837ecf1f
github-actions
Bot
added
team-Remote-Exec
iancha1992
enabled auto-merge
May 21, 2026 18:37
tyler-french
approved these changes
May 21, 2026
Wyverald
approved these changes
May 22, 2026
iancha1992
added this pull request to the merge queue
May 22, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
May 22, 2026
iancha1992
added this pull request to the merge queue
May 22, 2026
Merged
via the queue into
bazelbuild:release-9.1.1
with commit May 22, 2026
738967f
46 checks passed
github-actions
Bot
removed
the
awaiting-review
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Jun 9, 2026
### What does this PR do? Bump `.bazelversion` from 9.1.0 to 9.1.1. ### Motivation 9.1.1 ships several fixes in the Remote Execution section, some of which are relevant to remote caching: - bazelbuild/bazel#29583, - bazelbuild/bazel#29593, - bazelbuild/bazel#29596, - bazelbuild/bazel#29614, - bazelbuild/bazel#29616 May affect us: `.adms/bazel/bazel.bazelrc` sets this flag, so the 9.1.0 regression could have caused two configs to stack silently, potentially bypassing the ADMS mirror, - bazelbuild/bazel#29671, - bazelbuild/bazel#29677. ### Describe how you validated your changes CI. ### Additional Notes See: https://github.com/bazelbuild/bazel/releases/tag/9.1.1
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Jun 9, 2026
### What does this PR do? Bump `.bazelversion` from 9.1.0 to 9.1.1. ### Motivation 9.1.1 ships several fixes in the Remote Execution section, some of which are relevant to remote caching: - bazelbuild/bazel#29583, - bazelbuild/bazel#29593, - bazelbuild/bazel#29614, - bazelbuild/bazel#29616 - `.adms/bazel/bazel.bazelrc` sets this flag, so the 9.1.0 regression could have caused two configs to stack silently (i.e. potentially bypassing the ADMS mirror), - bazelbuild/bazel#29677. ### Describe how you validated your changes CI. ### Additional Notes See: https://github.com/bazelbuild/bazel/releases/tag/9.1.1 Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
Lewis-E
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Jun 12, 2026
### What does this PR do? Bump `.bazelversion` from 9.1.0 to 9.1.1. ### Motivation 9.1.1 ships several fixes in the Remote Execution section, some of which are relevant to remote caching: - bazelbuild/bazel#29583, - bazelbuild/bazel#29593, - bazelbuild/bazel#29614, - bazelbuild/bazel#29616 - `.adms/bazel/bazel.bazelrc` sets this flag, so the 9.1.0 regression could have caused two configs to stack silently (i.e. potentially bypassing the ADMS mirror), - bazelbuild/bazel#29677. ### Describe how you validated your changes CI. ### Additional Notes See: https://github.com/bazelbuild/bazel/releases/tag/9.1.1 Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ld/bazel/pull/29341)
For
--experimental_remote_cache_chunkingimplemented in #28437This PR enables parallel uploads and downloads for chunked files to improve performance. Transport-level concurrency is still bounded by the existing remote cache gRPC connection/concurrency limits. The chunk transfer managers add a fixed per-blob window of 16 to prevent a single large blob from fanning out too aggressively.
To avoid issues with batches, uploads and downloads use simple sliding-window style transfer managers.
RELNOTES: CDC chunk uploads and downloads can now happen in parallel within a large blob.
Benchmarks were rerun on 2026-05-12 with the JMH target
//src/test/java/com/google/devtools/build/lib/remote:ChunkedTransferBenchmark. The parent baseline was measured by running the same benchmark harness against the parent commit.With the synthetic benchmark of network delays and simulated jitter, the current branch is about 11-13x faster than the parent baseline for these cases. As usual, this synthetic benchmark is not a substitute for real remote-cache measurements.
After Change:
Before Change:
Big File:
Closes #29341.
PiperOrigin-RevId: 919144974
Change-Id: Iaa0ca8971bd21c879f21c708327b5ddd837ecf1f
Description
Motivation
Build API Changes
No
Checklist
Release Notes
RELNOTES: None
Commit 084958b