Skip to content

feat(storage): implement atomic compose via DeleteSourceObjects#4804

Draft
PranjalC100 wants to merge 4 commits into
masterfrom
feature/delete-source-objects-compose
Draft

feat(storage): implement atomic compose via DeleteSourceObjects#4804
PranjalC100 wants to merge 4 commits into
masterfrom
feature/delete-source-objects-compose

Conversation

@PranjalC100

@PranjalC100 PranjalC100 commented Jun 26, 2026

Copy link
Copy Markdown
Member

Description

This pull request implements an atomic compose operation by introducing a DeleteSourceObjects boolean inside the ComposeObjectsRequest. It leverages the new GCS Compose API functionality that allows atomically deleting source objects on the server-side the exact millisecond a compose operation completes.

This change fundamentally resolves race conditions, early deletion billing spikes, and "soft delete" charges commonly encountered during append-heavy AI/ML workloads.

Key Changes:

  • Dependency Upgrade: Upgraded cloud.google.com/go/storage to v1.63.0 to gain access to the DeleteSourceObjects attribute on the Compose request.
  • Core Implementation: Passed the DeleteSourceObjects: true flag during GCS object composition via ComposeObjectCreator and removed the manual DeleteObject deferred calls.
  • Garbage Collector (Fallback): The background garbage collector was intentionally retained. If a ComposeObjects API call fails mid-flight, the server-side deletion never triggers. The unmodified garbage collector acts as an excellent, zero-overhead resilient fallback to find and delete these rare orphaned chunks.
  • Mock Updates: Updated the ComposeObjects signature in fake/bucket.go to explicitly simulate atomic slice deletion of source objects inside integration test mocks.

Link to the issue in case of a bug fix:

b/430606937

Testing details

  1. Manual - Executed standard test sweeps and compilation checks locally via make build.
  2. Unit tests - Confirmed compose_object_creator_test.go properly asserts for req.DeleteSourceObjects and that legacy DeleteObject expectations were removed. Added a new explicit DeleteSourceObjects unit test suite to internal/storage/fake/testing/bucket_tests.go. All tests passed successfully.
  3. Integration tests - NA

Any backward incompatible change? If so, please explain.

N/A

@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.68%. Comparing base (b609bb2) to head (482c0ab).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4804       +/-   ##
===========================================
+ Coverage        0   83.68%   +83.68%     
===========================================
  Files           0      168      +168     
  Lines           0    20897    +20897     
===========================================
+ Hits            0    17487    +17487     
- Misses          0     2758     +2758     
- Partials        0      652      +652     
Flag Coverage Δ
unittests 83.68% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant