Standalone: delete snapshotted history by default in new data directories

[cloutiertyler]

Description of Changes

Closes #5542.

SpacetimeDB Standalone previously kept the entire commitlog and all snapshots on disk forever, so the data directory grew without bound. As @gefjon explained on the issue, commitlog segments whose entire contents precede the most recent snapshot are not needed to restart the database, and neither are older snapshots. SpacetimeDB-cloud archives this historical data to cold storage, but standalone had no lifecycle management at all.

This PR adds a retention policy for historical data. Deletion is the default for newly created data directories, while existing data directories keep their history on upgrade.

Behavior

A new background task (spawned by LocalPersistenceProvider, replacing the commitlog compressor when deletion is enabled) runs once at startup and then whenever a new snapshot is created. It:

1. retains the retain-snapshots most recent snapshots (default 2) and deletes all older ones, and
2. deletes all commitlog segments whose entire contents precede the oldest retained snapshot, using the same segment-boundary arithmetic as the existing snapshot_watching_commitlog_compressor (a segment is named for the first transaction it contains, so the last segment starting at or before the snapshot offset is kept).

The commitlog is retained from the oldest retained snapshot onwards, so the database can be restored from any retained snapshot even if the newest one turns out to be unreadable. The initial run at startup means a deployment that enables deletion reclaims disk space at the next restart without waiting for the next snapshot.

Because snapshots are captured from committed, not necessarily durable, state, and restore only considers snapshots at or below the durable commitlog offset, the pruner ignores snapshots above the durable offset entirely: they neither count towards the retention limit nor are they ever deleted. This mirrors the "decided snapshot" gating that cloud archival applies, and guarantees a restorable snapshot plus its commitlog suffix always survive a crash.

Snapshot deletion renames the directory to .archived_snapshot before removing it (the same mechanism cloud archival uses), so a prune interrupted mid-deletion can never leave a partially deleted directory that looks like a valid snapshot; leftovers are swept on the next run. When deletion is enabled, local snapshot compression is disabled, since compressing snapshots that will be deleted after the next snapshot is wasted work.

Configuration

A new [retention] section in the standalone config.toml, documented in the config template and in the standalone configuration reference docs:

[retention]
# "delete": delete historical data whenever a new snapshot is taken.
# "keep":   keep historical data on disk indefinitely; historical commitlog
#           segments are compressed (the previous behavior).
policy = "delete"

# Number of most recent snapshots to retain when policy = "delete".
retain-snapshots = 2

The default is applied through the config.toml template, which standalone writes into every newly created data directory and which sets policy = "delete". A config without a [retention] section behaves as policy = "keep", so data directories created by earlier versions keep their history when the server is upgraded; their operators can opt in by adding the section. The Rust-level default of RetentionPolicy is Keep accordingly.

Cloud is unaffected, as it does not use LocalPersistenceProvider outside of tests.

Implementation notes

• spacetimedb-snapshot: new SnapshotRepository::nth_latest_snapshot (computes the retention cutoff) and SnapshotRepository::prune_snapshots_before (deletes snapshots strictly before the cutoff, renaming before removal so a partially deleted snapshot is never mistaken for a valid one, and sweeping leftovers of interrupted runs). These are shared with the cloud archiver's delete policy in the companion private PR, replacing its previously duplicated logic.
• spacetimedb-commitlog: new Commitlog::remove_segments, mirroring compress_segments. It refuses to remove the writable head segment, removes oldest first so a failure partway through cannot leave a gap in the log, and keeps the in-memory segment list consistent. Also a new segments_older_than helper selecting the segments whose entire contents precede a transaction offset, now used by the compressor task, the pruner, and the cloud archiver.
• spacetimedb-durability: Local::remove_segments delegating to the commitlog.
• spacetimedb-engine: RetentionConfig/RetentionPolicy next to the existing DurabilityConfig, plus prune_history and the snapshot_watching_history_pruner task in relational_db, built on the helpers above.
• spacetimedb-standalone: config plumbing, config template, and docs.

API and ABI breaking changes

None. StandaloneOptions gains a retention field.

Expected complexity level and risk

3. This deletes user data by design, so the risk is in deleting too much. Mitigations: deletion is only enabled by explicit config (written into new data directories by the template, absent in pre-existing ones), the pruner only acts when more than retain-snapshots durable snapshots exist, never touches the segment containing the oldest retained snapshot or anything after it, never touches snapshots above the durable offset, and the head segment cannot be removed at the commitlog layer.

Testing

• New remove_segments unit test in the commitlog crate: refuses the writable segment, removes exactly the requested segments, and the log remains traversable afterwards.
• New prune_history test in the engine crate against a real fs-backed Local durability with tiny segments: no-op below the retention threshold, deletes exactly the stale snapshots and segments, cleans up leftover .archived_snapshot directories, ignores snapshots above the durable offset, and the remaining commitlog covers the oldest retained snapshot through the latest transaction.
• Standalone config tests: [retention] parses; the shipped config.toml template enables deletion; a config without the section keeps history.
• Full test suites of the commitlog, durability, engine, and standalone crates pass; clippy is clean.

[kdletters]

Could i manully trigger stdb to create a new snapshot?

[cloutiertyler]

@kdletters Not currently, no. There is no CLI command or HTTP endpoint to request a snapshot manually. Snapshots are taken automatically in two situations:

1. whenever a commitlog segment fills up and rotates, which is every 1GiB of commitlog by default and can be tuned via max-segment-size in the [commitlog] section of the standalone config.toml, and
2. every 1,000,000 transactions (SNAPSHOT_FREQUENCY, which is not currently configurable).

If your goal is more frequent snapshots, for example so that the delete retention policy from this PR reclaims disk sooner, lowering commitlog.max-segment-size is the practical knob today: each rotation requests a snapshot, and the pruner runs after every snapshot (and once at startup).

A manual trigger, e.g. a spacetime CLI subcommand or an admin endpoint, seems like a reasonable feature request. Please feel free to submit a feature request here: https://spacetimedb.com/features.