perf: reduce DB::view capture cost and COW reclamation debt

[feichai0017]

What changed

Reduce the cost of consistent multi-tree reads and prevent short-lived COW versions from becoming unnecessary checkpoint/store debt, without changing the public API, on-disk format, cross-tree consistency, escaped-reader lifetime or WAL/fsync acknowledgement boundary.

• Deduplicate DB::view scopes (preserving last-prefix-wins), flush frozen-tree deltas once, copy roots without redundant zeroing and batch-register independent epoch leases.
• Summarize named-tree snapshot epochs on the catalog root at checkpoint, including clean/truncate decisions and exhausted-clock reopen handling, instead of dirtying every captured root on reads.
• Track each detached image's creation-to-detachment visibility interval. Index its waiters by blocking epoch, so newer snapshots do not keep older versions alive. GC root pins also retain epoch protection.
• Bound cancellation of unreachable, never-persisted images. Preserve WAL low-watermarks in successors/promoted parents and publish child debt before releasing its write latch. Serialize cancellation with complete dirty capture; retain the entire dependency frontier while any older flushing claim exists. Persisted GUIDs still use safe durable reclamation, never inline disk deletion.
• Add lifecycle, epoch-exhaustion, foreign-tree COW/checkpoint, and sync-WAL recovery-without-final-checkpoint regressions. Include an independent, unpublished agent-workspace Holt/SQLite driver and reproducible balanced A/B runner in a separate commit.

Test plan

Local checks on the branch rebased onto main 36fb295:

• cargo fmt --all --check, plus formatting of the independent benchmark crate.
• cargo clippy --offline --locked --workspace --all-targets --all-features -- -D warnings.
• Workspace all-features doctests and RUSTDOCFLAGS="-D warnings" cargo doc --offline --locked --workspace --all-features --no-deps.
• cargo check --offline --locked --manifest-path benches/agent_workspace/Cargo.toml --target-dir target and both runner CLI checks.
• Final cargo test --locked --workspace --all-features --lib --tests --examples --quiet rerun passed, including all 493 library unit tests, integration/example targets and workspace binding tests. Existing opt-in ignored tests remain ignored.

The measured implementation previously passed the complete lib/integration/example suite, a 100k-operation DB oracle soak, and a 30-second / 95-round SIGKILL/reopen smoke for acknowledged cross-tree writes. Additional background correctness runs (100k metadata, a different-seed 40k metadata, and 12k content) reopened successfully without background errors. These are smoke tests, not power-cut or long-running qualification. Linux/io_uring, fuzz, coverage and MSRV remain CI gates.

Performance evidence

Full report and caveats. Apple M3 Pro; the identical handler/configuration in 132 balanced baseline/patched/SQLite processes plus 12 sync follow-ups. All 144 passed complete ordered-domain fingerprints and reopen checks.

Bounded workload	Old Holt	Patched Holt
Metadata async, 4 clients, 12k requests, no background checkpoint	226 ops/s	5,826 ops/s
Same run: peak RSS	6,800 MiB	107 MiB
Same run: allocated metadata after drain	8,160 MiB	40.7 MiB
Same run: final checkpoint	12,620 ms	70.1 ms
Content async, 4 clients, 6k requests, no background checkpoint	761 ops/s	4,593 ops/s

Single-client metadata view+reads stage decreased about 18% (not a pure-capture microbenchmark). Sync workloads were broadly unchanged: attached-metadata initially measured -3.0%, then -0.24% on balanced replication, with overlapping process ranges; this is not a universal zero-regression claim.

Numbers were measured against f080daf plus the recorded patch, before rebasing over main's manifest dual-writer fix and dependency update. They are not a fresh full performance matrix of the rebased head. SQLite OFF is diagnostic, not crash-equivalent; native background checkpoint cadence is not matched. Burst gains are not steady-state claims. Four-client metadata remains slower than SQLite OFF; root capture still copies one 512 KiB frame per unique tree. A longer correctness-only run retained 76 reclaimable GUIDs after one bounded cleanup, so universal backlog elimination is not claimed.

Only aggregate results, methods and bundle hashes are committed. Large raw samples, exact-source bundles and unredacted desktop process snapshots are retained locally, not published.

Related

Follow-up to the agent-workspace comparison on Holt 0.9.2. No version bump, release, NoKV OCC changes, or business history/outbox retention changes. Two logical commits: engine/tests/design, then reproducible benchmark/evidence.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.