Add BlobStore batch operations#28
Merged
Merged
Conversation
TJbrunk
approved these changes
Jul 1, 2026
TJbrunk
left a comment
Contributor
There was a problem hiding this comment.
The python sdk doesn't use any FFI bindings from this repo. It imports the rust code directly and creates it own bindings during compilation. Don't need to delete them, just an FYI.
TJbrunk
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the
integritySDK companion change for GOV-342. It adds batch blob-store operations at the trait boundary so callers that register or resolve many lineage blobs can avoid one fully serial blob-store request per CID. The default implementation runs existing single-object operations with bounded async concurrency, while leaving room for provider-specific overrides where a backend supports a native batch API.The PR also exposes the new batch operations through the FFI layer used by the Python SDK, so downstream services can call the SDK batch path without waiting on the CLI/core-rs path.
Problem
Lineage registration and resolution can involve many blobs, especially when an Iroh collection references multiple file blobs. The existing
BlobStoreAPI only exposedexists,get, andput, so every caller had to issue individual blob operations. In service paths this adds avoidable latency from serial network round trips and prevents storage backends from optimizing at their own layer.S3 existence checks were also doing paginated object listing to determine whether a single CID existed, which is much more expensive than a direct object metadata check.
Changes
BlobPut,BlobPutResult,BlobGetResult, andBlobExistsResulttypes.BlobStore::exists_many,BlobStore::get_many, andBlobStore::put_manydefault methods.existsto useHeadObjectagainst the exact object key instead of paginatedListObjectsV2.ig_blob_store_exists_manyig_blob_store_get_manyig_blob_store_put_manyget_many.Validation
cargo fmt --allcargo check -p integrity-blob --features blob-allcargo test -p integrity-blobcargo test -p integrity-fficargo test -p integrity-lineage-modelscargo check -p integrity --no-default-features --features blob-all,lineagegit diff --checkFollow-up
After this merges, we should mint a new
integritySDK version and update theintegrity-service/Python SDK dependency to consume these FFI batch APIs. Native provider batch overrides can be added later where a backend supports them directly.