Skip to content

Add BlobStore batch operations#28

Merged
captjt merged 3 commits into
mainfrom
gov-342-blobstore-batch-performance
Jul 2, 2026
Merged

Add BlobStore batch operations#28
captjt merged 3 commits into
mainfrom
gov-342-blobstore-batch-performance

Conversation

@captjt

@captjt captjt commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

This is the integrity SDK companion change for GOV-342. It adds batch blob-store operations at the trait boundary so callers that register or resolve many lineage blobs can avoid one fully serial blob-store request per CID. The default implementation runs existing single-object operations with bounded async concurrency, while leaving room for provider-specific overrides where a backend supports a native batch API.

The PR also exposes the new batch operations through the FFI layer used by the Python SDK, so downstream services can call the SDK batch path without waiting on the CLI/core-rs path.

Problem

Lineage registration and resolution can involve many blobs, especially when an Iroh collection references multiple file blobs. The existing BlobStore API only exposed exists, get, and put, so every caller had to issue individual blob operations. In service paths this adds avoidable latency from serial network round trips and prevents storage backends from optimizing at their own layer.

S3 existence checks were also doing paginated object listing to determine whether a single CID existed, which is much more expensive than a direct object metadata check.

Changes

  • Added BlobPut, BlobPutResult, BlobGetResult, and BlobExistsResult types.
  • Added BlobStore::exists_many, BlobStore::get_many, and BlobStore::put_many default methods.
  • Added a bounded default concurrency limit for batch methods so all current providers immediately benefit without provider rewrites.
  • Updated S3 exists to use HeadObject against the exact object key instead of paginated ListObjectsV2.
  • Added FFI batch APIs:
    • ig_blob_store_exists_many
    • ig_blob_store_get_many
    • ig_blob_store_put_many
    • matching result-free helpers
  • Updated manifest blob resolution so Iroh collection file blobs are fetched through get_many.
  • Added tests for default batch method ordering/missing-blob behavior and FFI local filesystem batch roundtrips.

Validation

  • cargo fmt --all
  • cargo check -p integrity-blob --features blob-all
  • cargo test -p integrity-blob
  • cargo test -p integrity-ffi
  • cargo test -p integrity-lineage-models
  • cargo check -p integrity --no-default-features --features blob-all,lineage
  • git diff --check

Follow-up

After this merges, we should mint a new integrity SDK version and update the integrity-service/Python SDK dependency to consume these FFI batch APIs. Native provider batch overrides can be added later where a backend supports them directly.

@captjt captjt marked this pull request as ready for review July 1, 2026 16:33
@captjt captjt changed the title [codex] Add BlobStore batch operations Add BlobStore batch operations Jul 1, 2026
@captjt captjt requested a review from TJbrunk July 1, 2026 16:33
@captjt captjt self-assigned this Jul 1, 2026

@TJbrunk TJbrunk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The python sdk doesn't use any FFI bindings from this repo. It imports the rust code directly and creates it own bindings during compilation. Don't need to delete them, just an FYI.

@captjt captjt requested a review from TJbrunk July 1, 2026 20:06
@captjt captjt merged commit a3f6e93 into main Jul 2, 2026
3 checks passed
@captjt captjt deleted the gov-342-blobstore-batch-performance branch July 2, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants