Support streaming split/splice APIs for large blobs

[tyler-french]

This PR adds streaming versions of split and splice. This lets clients send the ordered chunk list across multiple messages instead of fitting it all into one unary request or response.

Streaming also gives servers a chance to process the mapping as it arrives. For splice, this can avoid doing all of the work in one final RPC after every chunk has already been uploaded. Servers can still choose to wait until the stream is complete and process it like the unary path.

This solves the issues described in #376.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds streamed equivalents of CAS blob split/splice operations to avoid unary message size limits while exposing capability flags for feature discovery.

Changes:

• Document unary SplitBlob/SpliceBlob size limitations and recommend streaming alternatives.
• Add SplitChunks (server-streaming) and SpliceChunks (client-streaming) RPCs plus request/response messages.
• Extend CacheCapabilities with support flags for the new streaming RPCs.

Files not reviewed (2)

• build/bazel/remote/execution/v2/remote_execution.pb.go: Language not supported
• build/bazel/remote/execution/v2/remote_execution_grpc.pb.go: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[EdSchouten]

I got added as a reviewer, so here's my take: just like how chunking has been added to the protocol in its current form, I'm not a fan of this. Not having streaming, but requiring a recursive construction of very large files using Prolly trees is in my opinion the way to go.

That said, if people actually on the working group think that this is the way to go, go for it! :-)

[tyler-french]

I got added as a reviewer, so here's my take: just like how chunking has been added to the protocol in its current form, I'm not a fan of this. Not having streaming, but requiring a recursive construction of very large files using Prolly trees is in my opinion the way to go.

That said, if people actually on the working group think that this is the way to go, go for it! :-)

@EdSchouten Thanks, that makes sense. I agree that a recursive/Merkle-style representation such as Prolly trees is probably the better long-term model for representing extremely large blobs, since it bounds individual node sizes and avoids one giant flat chunk list.

My goal with this PR is narrower: provide a transport-level alternative to #376 without making clients materialize or dereference a chunk-list object in CAS. Streaming lets clients send the existing ordered chunk mapping in bounded batches, and it lets servers begin verification as chunks become available instead of waiting until every chunk has been uploaded and then receiving the whole mapping in one unary SpliceBlob request.

I also see this as compatible with a future recursive representation. A server could still store the received mapping internally as a Prolly tree or some other recursive structure. The protocol question here is whether clients need to understand and construct that representation, or whether they can just stream the chunk references they already have.

Longer term, I would prefer for the streaming APIs to become the primary split/splice shape for clients that support them, with the existing unary APIs remaining as the compatibility/simple-case path. For cases where the chunk list fits in one request, sending it as a single stream message should be performance-equivalent to a unary gRPC request in practice, while still leaving the same API able to scale when the list grows. A recursive representation also fits into that model: the stream does not require the server to store a flat list internally, it just gives the server live visibility into the composition progress. The server can choose to persist that mapping as a Prolly tree, a flat manifest, or another internal structure, while still getting the benefits of streaming transport: bounded message sizes, earlier validation, and the ability to pipeline verification while the client is still uploading or discovering chunks, since chunking is inherently sequential.

Some recursive composition can be approximated with the current API today by splicing subranges and then splicing those intermediate blobs, but standardizing that as the preferred model seems like it would need more protocol work: tree node encoding, fanout rules, digest/validation semantics, and how chunking_function applies at each level. Streaming is a smaller additive step that addresses the immediate unary message-size and final-RPC verification problem while leaving room for that design.

[tyler-french]

@EdSchouten One open question I have for a Prolly tree design is how it preserves the existing full-blob digest semantics. In REAPI, the digest in an ActionResult is the digest of the output file contents, and CAS lookups are keyed by that full content digest. So even if the server stores the blob as a recursive tree internally, both the client and server eventually need to know that the tree represents exactly the bytes for the full blob digest.

If the Prolly root digest is a digest of tree-node bytes, it is not a substitute for the file's content digest. If the key remains the full blob digest, the server still has to verify the leaves in order by hashing the reconstructed content, or trust a mapping it has already validated. And if a server only has chunks/tree nodes under one chunking strategy, it is not obvious how it can know that a different client-provided tree already represents the same blob without either having a stored full-digest-to-tree mapping or walking/rehashing the content.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 3 changed files in this pull request and generated 2 comments.

Files not reviewed (2)

• build/bazel/remote/execution/v2/remote_execution.pb.go: Language not supported
• build/bazel/remote/execution/v2/remote_execution_grpc.pb.go: Language not supported

[moroten]

The streaming approach keeps chunk-list transport in the RPC layer instead. That has a few advantages:

• It avoids using CAS as a protocol transport for RPC metadata.

That is true for this specific RPC, but many other parts of the protocol already work this way.

• It does not require clients to add extra read/write logic for chunk-list objects.

I agree that it would probably be easier to adapt most servers and clients.

• It lets servers receive the ordered chunk mapping incrementally instead of only at the final splice call.

The speed improvement I can see is that splicing can finish faster after the last chunk has been uploaded. The work still has to be done, so it is a matter of cutting latency, instead of loading GB of data at the end, it can be loaded while the client is uploading the chunks. @tyler-french Do you see the latency as a problem in your current clusters?

My biggest concern is that the servers may have to hold an infinite number of connections open, waiting for more chunks to arrive.

• What timeout should be allowed? For Bazel, we will still have the be quicker than the global --remote_timeout.
• If a splice call times out, the server has to throw away all work. With a unary call with a chunk-list the server can process the whole splicing and store the result, so timeout should not be any problem (unless server internal networking is slow).
• Minor note: For clients to handle timeouts, they need to keep the full list of chunks until splicing is done. There will not be any benefit in lower memory consumption on client side.
• With a unary call, the splicing can be deduplicated on server side.

To address my concerns, I might implement the streaming splicing by waiting for the last message and then do the splicing anyway, effectively converting it into the chunk-list version.

• It keeps the old unary APIs simple and fully usable by older clients.

The chunk-list approach also keeps the old API fully usable by older clients.

• It gives clients a capability-gated path for large blobs without changing the semantics of existing fields.

Either we have the feature of handling large blobs capability gated or we can put it as requirement on Split/Splice API from a certain version of the REv2 API.

[tyler-french]

@moroten (and cc @meroton-benjamin) Thanks, these are good concerns. I am also happy to support #376 if that is where we land. I needed to add streaming support on our side anyway, so this felt like a good time to see whether it could also solve the large chunk-list problem.

I think the main point is that streaming does not remove the splice work. It gives clients a bounded-message way to send the chunk list, and it lets servers start seeing the mapping earlier if they want to. A client does not have to keep the stream open while chunks are still uploading. It can upload the chunks first, then call SpliceChunks with the full chunk list in one or a few messages. That makes this usable as an alternative to the indirect chunk-list approach, without requiring the chunk list itself to be stored and fetched through CAS.

The speed improvement I can see is that splicing can finish faster after the last chunk has been uploaded. The work still has to be done, so it is a matter of cutting latency, instead of loading GB of data at the end, it can be loaded while the client is uploading the chunks. @tyler-french Do you see the latency as a problem in your current clusters?

Yes. We have hit issues with blobs around 1 GB. Today the client uploads chunks with BatchUpdateBlobs, then calls SpliceBlob after all chunks are available. Since the server only sees the mapping at the end, all splice work is deferred into one final RPC, and we have seen that call time out.

I agree that the work still has to be done. The benefit is mostly message sizing and tail latency. With a fast chunker, I would expect an incremental SpliceChunks flow to take about as long as Write for the same blob.

My biggest concern is that the servers may have to hold an infinite number of connections open, waiting for more chunks to arrive.

That risk is real, but it is similar to Write. Servers still need normal RPC controls, such as deadlines, idle timeouts, max concurrent streams, and per instance limits.

Also, clients that want to avoid long idle splice streams can wait until all chunks are uploaded, then send the mapping in one or a few SpliceChunks messages. That gives a similar call shape to PR #376, but keeps the chunk-list transport in the RPC layer.

What timeout should be allowed? For Bazel, we will still have the be quicker than the global --remote_timeout.

I think this should follow the normal RPC deadline model. For Bazel, the call still needs to complete inside --remote_timeout. Probably something similar to whatever the Write call was for the blob.

If a splice call times out, the server has to throw away all work. With a unary call with a chunk-list the server can process the whole splicing and store the result, so timeout should not be any problem (unless server internal networking is slow).

This is the main tradeoff. If the stream fails before the server commits the splice, the client has to retry. After the chunks are uploaded, that retry is much cheaper than retrying the original upload.

There is also a useful fallback. If the chunk list fits in one message, the client can retry with unary SpliceBlob. If it does not, it can retry SpliceChunks with smaller batches or only a few messages. So the client has a bit more control over getting things to succeed

Minor note: For clients to handle timeouts, they need to keep the full list of chunks until splicing is done. There will not be any benefit in lower memory consumption on client side.

Agreed. Keeping a []digests is pretty cheap. You could even just keep []offsets to save even more memory.

With a unary call, the splicing can be deduplicated on server side.

I got this feedback, and I think it makes sense the request requires blob_digest. It should deduplicate well now by routing by this.

To address my concerns, I might implement the streaming splicing by waiting for the last message and then do the splicing anyway, effectively converting it into the chunk-list version.

That seems like a valid implementation. It would still avoid the unary message size limit and let clients use one API shape for larger blobs.

The chunk-list approach also keeps the old API fully usable by older clients.

Agreed. I did not mean that the chunk-list approach breaks older clients.

The difference I see is that PR #376 adds another representation to the existing unary API shape. Newer clients then need logic to write, read, and interpret chunk-list objects. With streaming, the old unary APIs stay as the simple path, and the large-blob path is a separate capability-gated RPC. A client that is not aware of indirection may fail if the SplitBlob response sets it.

[moroten]

@tyler-french I would be happy with the streaming approach as well. Both approaches have their pros and cons, but the main problem is that the whole API refers to blobs as whole files, not a Prolly tree.

Maybe set the decision next monthly? I'll add some code comments as well.

[tyler-french]

LGTM, but let's make sure feedback from other reviewers has been addressed (please mark conversations resolved as appropriate).

Thanks @tjgq! I addressed all previous feedback. @sluongng @fmeum @vanja-p do you all have time to take a look?

[sluongng]

small doc nits, but lgtm

[tyler-french]

Thinking about this more: if a client wants to start uploading and hashing at the same time, the blob hash shouldn't be required in the first request - it just needs to be in some request, or the last one.

Routing isn't important here because the chunks will exist on many shards, so only the final frame needs routing to know where to store the verified entry.

Going to update to account for this

[tyler-french]

Just made in important change, that will further make this more performant:

Updated SpliceChunks so blob_digest is required only on the final request. A client may not know the full blob digest when it begins chunking, so this allows chunking, uploading, and incremental splice verification to proceed concurrently.

This does not weaken integrity: each chunk is content-addressed, and the server incrementally hashes the concatenated chunk contents in order, then compares the result with blob_digest when committing.

Earlier requests do not need the blob digest for routing because each chunk is routed independently by its own digest. Only the final commit needs blob_digest to validate and route storage of the resulting manifest.

[moroten]

Updated SpliceChunks so blob_digest is required only on the final request. A client may not know the full blob digest when it begins chunking, so this allows chunking, uploading, and incremental splice verification to proceed concurrently.

How does the client know that it does need to upload the blob at all? I thought FindMissingBlobs would be called on the full blob before calling SpliceChunks but this indicates that the recommended process is to split the blob and stream the chunk digests. Before sending a chunk digest, it first must go through FindMissingBlobs for potential upload.

My original though was to first run FindMissingBlobs on the digest of the full blob before starting SpliceChunks.

Does calling many FindMissingBlobs really increase the efficiency compared to #376?

Earlier requests do not need the blob digest for routing because each chunk is routed independently by its own digest. Only the final commit needs blob_digest to validate and route storage of the resulting manifest.

Well, that depends on the implementation of your server. Some servers might want to shard out the splicing process depending on the final blob_digest.

With the blob_digest up front, concurrent uploads can be aborted with ALREADY_EXISTS once the first upload finished.

[tyler-french]

@moroten good points. I moved blob_digest back to the first request. You're right that clients should generally call FMB for the full blob first, and sending the digest up front gives servers more control over routing and early deduplication. It could also short circuit if it it already has the blob but FMB never got called.

[meroton-benjamin]

Having implemented a client/server pair with this RPC I think we need to add guidance towards how this should be used to to upload a large blob. The naïve implementation, where you calculate a chunk, check if it exists, uploads it if it doesn't exist and sends a message with the chunk to the streaming rpc is incredibly inefficient and splicing is very intensive server side.

In essence a well behaved client should do the following for a large blob:

Step 1:

• Calculate the blob digest and determine if the blob exists or not

Step 2:

• Calculate a N of chunks of the blob where N is as many as can be comfortably held by the client, if calculating from a situation where you can do random access (e.g. when uploading a blob from disk) this may be something like 1MB worth of digests.
• Determine which if any of the chunks are missing from the CAS with FMB
• Upload those missing chunks
• Repeat until all chunks exist in CAS

Step 3:

• From the chunk list calculated earlier split into safe message sizes (e.g. below 1MB) and stream the splice chunks call.

While the api formally will work for clients which don't follow the above recipe, in practice server implementations will struggle to be able to serve it. A naïve implementation from above would from a very modest 1GB blob of random data create the equivalent of 10000 unary rpc calls.

[meroton-benjamin]

Could we rename this into SplitIntoChunks as SplitChunks doesn't really make any sense.