fix(s3): delete unused blocks with bounded concurrency

[christophersherman]

Which issue(s) this PR fixes:

Issue longhorn/longhorn#8060

What this PR does / why we need it:

Block GC already knows the exact object path for every unreferenced block. However, the S3 driver's existing Remove method treats each path as a prefix: it issues ListObjectsV2, creates another SDK client, and then calls DeleteObject. GC repeats that sequence serially for every unused block.

This PR adds an optional ExactFileRemover capability and uses it for volume-backup and backing-image block GC:

• Feed known block paths lazily in batches of at most 65,536, bounding additional path memory to roughly 1 MiB of string descriptors.
• Create one S3 SDK client per batch and share it across eight bounded workers.
• Issue individual, exact-key DeleteObject requests without a preceding LIST.
• Attempt every key and every batch before returning a bounded aggregate error.
• Treat NoSuchKey as an idempotent success.
• Preserve the current serial Remove fallback for all drivers that do not opt in.
• Keep volume BlockCount unchanged if any deletion fails.

For N unused S3 blocks, this changes the deletion phase from approximately N prefix LISTs plus N serial DELETEs to N bounded-concurrent DELETEs. It does not change reference discovery or lock safety.

Synthetic benchmark: 100 exact deletions with 1 ms fake request latency, Apple M1 Max, three iterations:

Path	Time/op	Allocated/op	Allocs/op
1 worker	113.72 ms	43,080 B	709
8 workers	14.82 ms	45,576 B	721

This is about 7.7x lower elapsed time in the latency-bound deletion phase. The HTTP regression test separately verifies exactly one DELETE per path and no LIST or multi-object-delete request.

Special notes for your reviewer:

This deliberately does not use the S3 Multi-Object Delete API. Longhorn removed that path in backupstore#41 because Google Cloud Storage's S3 interoperability rejected it (longhorn/longhorn#1404). The implementation uses concurrent individual DeleteObject calls instead.

This is independent of and complementary to backupstore#313:

• fix(s3): use recursive listing to avoid excessive API calls #313 reduces the sharded block-enumeration LIST count.
• This PR removes the redundant per-deleted-block LIST and parallelizes exact deletion.

Deletion locks of the same type can overlap. The limit is therefore eight requests per GC operation, not a global rate limit. Coalescing duplicate GCs, changing lock scope, and optimizing retained-manifest scanning remain separate work.

Regression coverage includes:

• exact-remover selection and serial-driver fallback
• safe-path filtering for volume and backing-image GC
• volume metadata preservation after partial failure
• bounded concurrency and bounded queues
• all-key/all-batch attempt behavior after errors
• bounded aggregate error details
• missing-key idempotency without hiding NoSuchBucket
• exact S3 request method/path/count
• 65,536-key batch-memory bound

Validation:

• Focused root cleanup tests with -race: pass
• go test -race ./backupbackingimage: pass
• go test -race ./s3: pass
• Linux cross-compilation of ./...: pass
• Linux go vet ./...: pass
• Linux golangci-lint run ./...: 0 issues

The repository's current scripts/test omits root-package tests, so the new root cleanup tests were invoked explicitly. The full pre-existing root suite also has the unrelated TestInspectBackup failure described by #313; this change does not modify that path.

Additional documentation or context

Historical review precedent already recommended a fixed worker pool and noted that DeleteObjects could benefit from parallel deletion: backupstore#63.

[christophersherman]

@mantissahz This is the deletion-phase follow-up discussed on backupstore#313 for the priority/0 longhorn/longhorn#8060 path. It is independent of #313 and keeps GCS compatibility by using bounded individual DeleteObject calls rather than the multi-object API.

DCO is green, and the PR includes exact request-count tests, focused race coverage, bounded 65,536-key batching, an eight-worker concurrency test, and comparative benchmark results. Could you help route maintainer review and the upstream Build binaries workflow?

[Copilot]

Pull request overview

Adds efficient, bounded-concurrency S3 block garbage collection while preserving fallback behavior for other drivers.

Changes:

• Introduces batched exact-file removal.
• Adds concurrent exact-key S3 deletion with bounded errors.
• Covers volume and backing-image cleanup behavior comprehensively.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
s3/s3.go	Implements bounded concurrent removal.
s3/s3_service.go	Adds exact DeleteObject handling.
s3/s3_remove_test.go	Tests S3 removal behavior.
exact_file_remover.go	Defines batching capability.
exact_file_remover_test.go	Tests batching and errors.
deltablock.go	Uses exact removal for volume GC.
deltablock_cleanup_test.go	Tests volume cleanup paths.
backupbackingimage/backupbackingimage.go	Uses exact removal for backing-image GC.
backupbackingimage/cleanup_test.go	Tests backing-image cleanup paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.