[Store] Add reliable per-object deletion and Bucket GC for LOCAL_DISK

[zchuango]

Description

Refs #3220 .

This PR implements reliable per-object deletion and physical space reclamation for LOCAL_DISK replicas in the Bucket backend.

The implementation is submitted as one end-to-end change because object incarnation, stable storage identity, durable delete intents, holder-side tombstones, ACK durability, Bucket GC, and crash recovery jointly form the correctness boundary of the deletion lifecycle.

The changes are organized by subsystem in the implementation, description, and tests to keep the review tractable.

Currently, Remove and BatchRemove primarily remove logical metadata from the Master, but they do not reliably propagate per-object deletion to the remote Bucket backend holding the corresponding LOCAL_DISK replica.

This can result in:

• deleted objects being reloaded from old bucket metadata after holder restart;
• dead bytes remaining on SSD indefinitely;
• pending deletions being lost after primary failover;
• delayed tasks deleting a newly recreated object with the same tenant/key;
• SSD high-watermark handling being unable to reclaim dead bytes before evicting live data.

This PR adds the following end-to-end deletion path:

Remove / BatchRemove
        ↓
durable local-delete intent
        ↓
holder bounded Fetch
        ↓
incarnation-checked durable tombstone
        ↓
durable ACK
        ↓
background watermarked Bucket GC

Remove/BatchRemove does not wait for physical GC. A successful return means the logical deletion and the delete intent supported by the current deployment mode have been accepted; it does not mean SSD space has already been synchronously reclaimed.

What changed

1. Object incarnation and stable storage identity

• Generate an immutable ObjectIncarnation for every newly created logical object.

• Propagate the incarnation through:

  • Master and standby metadata;
  • snapshots;
  • OpLog;
  • offload tasks;
  • LOCAL_DISK descriptors;
  • bucket metadata;
  • local-delete tasks.

• Persist a stable LOCAL_DISK identity in:

  .mooncake_local_disk_segment_id
  

• Add stable local_disk_segment_id and per-owner mount_epoch.

• Validate client_id, storage identity, mount epoch, and delete capability during Fetch/Ack.

• Fence the old epoch when a new owner mounts the same storage identity.

• Use a local advisory lock to prevent two processes on the same node from opening the same directory concurrently.

• Fail closed by default for non-empty legacy directories without an identity marker.

• Allow explicit administrator adoption through:

  MC_STORE_ADOPT_UNMARKED_LOCAL_DISK=1
  

2. Master-side deletion protocol

• Create delete tasks for completed LOCAL_DISK replicas that advertise OBJECT_TOMBSTONE_V1.
• Add a bounded LocalDeleteRegistry with a capacity of 50,000 tasks.
• Reserve all required task slots before mutating the object state.
• Fail the corresponding Remove closed if reservation cannot be satisfied.
• Create at most one task for the same object incarnation on the same storage identity.
• Preserve per-key BatchRemove result semantics instead of introducing a batch-wide transaction.
• Fetch does not remove tasks and does not rely on ephemeral inflight state.
• Remove a pending task only after durable ACK.

3. Fetch/Ack RPC and HA recovery

• Add bounded FetchLocalDeleteTasks.
• Add bounded AckLocalDeleteTasks.
• Add a versioned REMOVE delete-intent payload.
• Add a versioned LOCAL_DELETE_ACK OpLog operation.
• Publish tasks only after the REMOVE durable callback.
• Remove tasks only after the ACK durable callback.
• Replay the same REMOVE intents and ACKs on standby.
• Persist pending tasks in primary and standby snapshots.
• Fail closed on corrupted or unknown new payload versions without advancing standby sequence state.
• Use at-least-once delivery plus idempotent processing for lost responses, holder restarts, and duplicate Fetches.

4. Holder-side durable tombstone

• Convert fetched user keys into tenant-scoped storage keys.
• Group fetched tasks by bucket.
• BatchMarkDeleted performs at most one metadata persistence operation per affected bucket.
• The persistence sequence is:
  1. copy bucket metadata;
  2. set tombstoned=true;
  3. write a temporary .meta;
  4. fsync the temporary file;
  5. rename it over the committed .meta;
  6. fsync the parent directory;
  7. remove the exact incarnation from the live index.
• ACK only terminal results:
  • Removed
  • AlreadyRemoved
  • StaleVersion
• Return RetryableFailure on file-write, fsync, rename, or other persistence failures.
• Prevent an incarnation mismatch from modifying a newly recreated object with the same key.
• Hide tombstoned objects from the holder live index, ScanMeta, BucketScan, and read paths.
• Run incarnation-aware reconciliation after holder startup, remount, or Master epoch change.

5. Watermarked Bucket GC

• Run one background GC worker per Bucket backend.
• Use a normal dead-byte ratio threshold of 25%.
• Wake the worker when new tombstones are created.
• When SSD usage reaches the existing high watermark:
  • bypass the normal ratio threshold;
  • prioritize buckets containing dead bytes;
  • continue reclaim toward the low watermark.
• Preserve the existing live-bucket eviction fallback when no reclaimable dead bytes remain.
• Unlink fully dead buckets without creating a replacement.
• Rewrite partially dead buckets using copy-on-write replacement.
• Merge at most 8 source buckets in one GC round.
• Keep replacements within existing bucket key/count and size limits.
• Stream live-record copies through a fixed 1 MiB buffer.
• Wait for old readers to drain before unlinking source files.
• Keep GC data copying off the Remove RPC and heartbeat critical paths.

The GC mechanism is implemented in this PR, while its final default rollout policy is intentionally left open for review. In particular, normal ratio-based GC and high-watermark-triggered reclaim can be configured differently if maintainers prefer a more conservative rollout.

6. GC crash recovery

• Persist local source/target transition state in:

  .bucket_gc_intent
  

• PREPARED:

  • sources remain authoritative;
  • uncommitted target is cleaned up.

• COMMITTED:

  • target is validated and retained;
  • source files are cleaned up.

• Fail closed if a committed target is missing or corrupted.

• Clean up stale bucket metadata during initialization when .meta exists but the corresponding data file is missing.

7. Snapshot and compatibility

• Upgrade the snapshot format from 1.0.0 to 1.1.0.

• Keep support for reading 1.0.0.

• Read legacy bucket metadata as:

  ObjectIncarnation = zero
  tombstoned = false
  

• Do not deliver delete tasks to backends that do not advertise the tombstone capability.

• Return SEGMENT_ALREADY_EXISTS for an identical repeated local-disk mount.

• Update mount state when the same client changes storage identity, epoch, or capability.

• Require participating components to understand the corresponding RPC, OpLog, and snapshot semantics before relying on the reliable-deletion guarantee.

The current implementation persists .mooncake_local_disk_segment_id for stable LOCAL_DISK identity.

For an existing non-empty directory without this marker, the current implementation fails closed unless explicit adoption is enabled through:

MC_STORE_ADOPT_UNMARKED_LOCAL_DISK=1

This behavior reflects the current implementation, but the final rollout policy for legacy unmarked directories is open to maintainer feedback. A compatibility mode for existing unmarked directories can be added if preferred.

Deployment semantics

Mode	Behavior
HA + OpLog	REMOVE intents and ACKs survive through OpLog/snapshot recovery
non-HA	pending tasks are in memory; Master crash relies on remount/reconciliation
HA without OpLog	reliable asynchronous physical deletion is not provided by the current implementation

Scope

This PR changes:

• Remove
• BatchRemove
• LOCAL_DISK tombstones in the Bucket backend
• watermarked Bucket GC
• related RPC, OpLog, snapshot, and compatibility metadata

This PR does not change:

• RemoveAll
• RemoveByRegex
• other storage backends
• the existing eviction policy
• the disconnected/grace state machine for warm re-adoption
• drain migration
• the general task scheduling framework

Review scope

This PR intentionally implements the complete LOCAL_DISK per-object deletion lifecycle in one change.

Although the implementation spans Master metadata, OpLog/snapshot recovery, holder-side storage handling, and Bucket GC, these components are coupled by the deletion correctness contract:

object identity
  → durable delete intent
  → durable local tombstone
  → durable ACK
  → physical reclamation

The PR is organized by subsystem so each component can be reviewed independently while preserving an end-to-end implementation that can be tested as a whole.

Known limitations

1. Permanently lost storage

A permanently lost storage identity does not automatically expire its pending tasks.

This is intentional fail-closed behavior because silently expiring tasks would weaken the deletion guarantee.

If the registry reaches capacity, new Remove operations may be rejected.

Explicit storage retirement, administrator force-drop, audit logging, and related metrics are left as follow-up work.

2. Runtime GC cleanup retry

If .bucket_gc_intent has already reached COMMITTED but source unlink, directory fsync, or reader drain fails or times out:

• the replacement target remains authoritative;
• deleted objects do not become visible again;
• physical source cleanup may remain incomplete;
• later reclaim may require holder restart and recovery.

This is a space-reclamation liveness limitation rather than an object-consistency failure.

Runtime retry of committed GC cleanup can be added as follow-up work.

3. Delete/GC relocation correctness

If GC relocates an object from a source bucket to a replacement bucket while deletion is in progress, the delete path must re-resolve the current bucket mapping and retry the tombstone against the replacement when the same incarnation is still present.

StaleVersion is only terminal when the object is truly absent or the current object belongs to a different incarnation.

This race is part of the correctness contract and should remain protected by deterministic regression coverage.

4. Performance validation

Bucket GC introduces additional live-byte copy I/O when reclaiming partially dead buckets.

The current implementation bounds this work through:

• one GC worker per Bucket backend;
• at most 8 source buckets per GC round;
• a fixed-size streaming copy buffer;
• existing bucket key/count and size limits;
• no GC data copying on the Remove/BatchRemove RPC path.

Representative SSD benchmarks for Get p99, GC throughput, write amplification, RSS, and foreground I/O interference are useful additional validation and are planned during review.

The final default GC policy can be adjusted based on maintainer feedback and benchmark results without changing the core reliable-deletion protocol.

Related work

• [Store] Fix RemoveAll not cleaning SSD offload files on any node via PollRemoveAll RPC #2676 handles whole-store cleanup for RemoveAll; this PR handles per-object deletion.
• [RFC]: Master-Side LOCAL_DISK Replica Warm Re-Adoption Across Client Restart #2306 and store: reattach local disk replicas after restart #2777 address LOCAL_DISK warm re-adoption. This PR shares stable identity and mount plumbing but does not implement a disconnected/grace period.
• If store: reattach local disk replicas after restart #2777 lands first, this branch will rebase and remove duplicated identity/mount implementation.
• If the identity foundation here lands first, store: reattach local disk replicas after restart #2777 may reuse the marker, mount epoch, and descriptor fields.
• [Store] Support LOCAL_DISK replica migration during drain job #3085 handles LOCAL_DISK replica migration during drain; source-disk cleanup may later reuse this task protocol.
• [Bug]: [SSD] DSV4 enable SSD offload，stress testing, repeat offloading the same batch key, lead to OBJECT_ALREADY_EXISTS/persist failed/INVALID_KEY #2827 covers same-key SSD races; incarnation prevents stale deletion from deleting a new object with the same key.
• If [Store] Split SSD unit tests #2901 lands first, the new tests will be rebased into the updated test organization.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

This is currently marked as a breaking/deployment compatibility change because the implementation introduces a persistent LOCAL_DISK identity marker and currently fails closed when mounting an existing non-empty directory without that marker.

Legacy snapshots and bucket metadata remain readable, and explicit adoption is supported.

The final behavior for legacy unmarked directories is still open to review. If a backward-compatible legacy mount mode is preferred and implemented, this PR may no longer need to be classified as a breaking change.

How Has This Been Tested?

Feature and regression tests

ctest --test-dir build-ssd-delete --output-on-failure -R   '^(master_service_test_for_snapshot|snapshot_child_process_test|master_snapshot_codec_test|storage_backend_test|batch_remove_test|master_service_ssd_test|file_storage_test|client_integration_test|non_ha_reconnect_test|master_service_ha_test|hot_standby_service_test|catalog_backed_snapshot_provider_test)$'

ctest --test-dir build-ssd-delete --output-on-failure -R   '^(storage_backend_bucket_delete_test|local_delete_test|oplog_applier_test|local_delete_process_kill_test)$'

ctest --test-dir build-ssd-delete --output-on-failure -R   '^(serializer_test|segment_test|oplog_types_test|master_service_test|master_service_ssd_test_for_snapshot)$'

The listed tests pass in the current development environment.

Current coverage includes:

• registry reservation and capacity;
• mount-epoch fencing;
• capability negotiation and wire compatibility;
• REMOVE/ACK standby replay;
• pending-task snapshot restore;
• primary SIGKILL before the REMOVE durable callback;
• ACK SIGKILL before persistence;
• durable tombstones;
• same-key recreation and stale incarnation handling;
• legacy metadata compatibility;
• fully dead bucket unlink;
• multi-bucket COW merge;
• SSD high/low-watermark reclaim;
• PREPARED/COMMITTED GC intent recovery;
• Master, FileStorage, SSD, HA, snapshot, serializer, segment, and client integration regressions.

Additional validation planned during review includes:

• a full multi-node Master → RPC → FileStorage → Bucket end-to-end deletion scenario;
• representative SSD latency and write-amplification benchmarks.

Test results:

• Unit tests pass
• Existing integration tests pass
• Additional multi-node end-to-end deletion validation
• Representative SSD latency / write-amplification benchmark

Checklist

• I have performed a human line-by-line review of every changed line
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on all touched files and all hooks pass
• I have updated the documentation
• I have added tests covering the primary feature and recovery paths
• For changes >500 LOC, an RFC has been filed
• This branch has been rebased onto the latest upstream main
• Overlap with upstream store: reattach local disk replicas after restart #2777 has been resolved or documented

AI Assistance Disclosure

• No AI tools were used
• AI tools were used

Codex was used to assist with design analysis, implementation, test construction, failure-path review, and documentation.

The submitter remains responsible for reviewing, understanding, testing, and defending the final implementation.