Skip to content

Add the TapDB backup and recovery lifecycle (#89) - #91

Open
jdurham38 wants to merge 1 commit into
jemdev10from
feat/backup-recovery-lifecycle
Open

Add the TapDB backup and recovery lifecycle (#89)#91
jdurham38 wants to merge 1 commit into
jemdev10from
feat/backup-recovery-lifecycle

Conversation

@jdurham38

@jdurham38 jdurham38 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Closes #89.

Replaces the previous ~140-line five-table backup command with a schema-scoped,
checksummed, manifest-described backup subsystem driven by one shared service
across CLI, admin API and the embedded GUI, so a fix to how backups are taken is
a fix everywhere at once.

What it delivers

Three backup classes (template-pack, full logical, provider-snapshot receipt).
Restores default to an isolated target; in-place requires a typed confirmation
and lands via a verified staged schema swap, so nothing is dropped until a
verified replacement exists and every failure path renames the original back.
Integrity is verified before and after, which is what makes EUID reuse
impossible. Every mutating run emits an immutable hash-chained receipt with a
head anchor, so truncation is detectable rather than merely unlikely.

All 10 issue items are complete

  • tapdb backup prune: retention enforcement. A backup is deletable only
    when its hold set is empty: nothing is ever selected for deletion, things
    fail to be protected. Fourteen holds behind seven gates that abort before a
    single byte is removed. Dry run is the default; deleting needs the typed target
    label, and a delete ceiling guards the case where that label has become a
    constant in a scheduled job.
  • tapdb backup health: the alerting contract. Exit 0 recoverable, 1
    something is wrong, 2 the check itself could not run, with parseable JSON on
    stdout on every code. The runbook documents cron/systemd/monitor wiring.

Please note

A live bug fix rode along. S3StorageBackend.list_keys built its Prefix
without a trailing delimiter, so listing acme/orders also returned
acme/orders-staging, inflating backup list, the status page and the GUI with
another target's backups. Unrelated to #89, S3-only, and invisible to existing
tests because local storage has a real directory boundary.

data backup / data restore are now deprecated in favour of tapdb backup.

Verification

1620 passing, 14 skipped. ruff, format and bandit clean. 91.5% coverage against
an 85% gate. Verified e2e against a real Aurora PostgreSQL 16.11 cluster.

Testing leaned on mutation testing throughout, deliberately breaking each
guarantee and confirming a test fails. That is what caught the defects a green
suite was hiding, including EUID reuse after in-place restore, CLI exit codes
reporting 0 for failures, and a prune reconciliation path that could erase an
entire store from a single unvalidated receipt field.

One thing

  • No tag yet. setuptools_scm already derives 9.0.11.dev1. Tag 9.0.11
    on the merge commit when something actually needs to pin it. 9.0.10 and every
    earlier tag are untouched, and no service pins a mutable ref, so nothing
    downstream moves.

Replaces the previous ~140-line five-table backup command with a
schema-scoped, checksummed, manifest-described backup subsystem driven by
one shared service across CLI, admin API and the embedded GUI.

Three backup classes (template-pack, full logical, provider-snapshot
receipt). Restores default to an isolated target; in-place requires a typed
confirmation and lands via a verified staged schema swap, so nothing is
dropped until a verified replacement exists and every failure path renames
the original back. Integrity is verified before and after, which is what
makes EUID reuse impossible. Every mutating run emits an immutable
hash-chained receipt with a head anchor, so truncation is detectable rather
than merely unlikely.

Item 10 is delivered in full:

* `tapdb backup prune` enforces retention. A backup is deletable only when
  its hold set is empty -- nothing is ever selected for deletion, things
  fail to be protected -- behind seven gates that abort before any byte is
  removed. Dry run is the default; deleting needs the typed target label,
  and a delete ceiling guards the case where that label has become a
  constant in a scheduled job.
* `tapdb backup health` is the alerting contract: exit 0 recoverable, 1
  something is wrong, 2 the check could not run, with parseable JSON on
  stdout on every code. It needs no database, so it still answers when the
  database is down, and it writes nothing, so it is safe to poll. Alert
  delivery is the consumer's -- an embedded library reaching out to a
  notification channel from inside another service's process would be a
  surprise, not a feature.

Also fixes a live defect found along the way: S3 list_keys built its Prefix
without a trailing delimiter, so listing acme/orders also returned
acme/orders-staging. Local storage has a real directory boundary, which is
why no existing test could see it.

Verification leaned on mutation testing throughout -- deliberately breaking
each guarantee and confirming a test fails. That caught defects a green
suite was hiding, including EUID reuse after in-place restore, CLI exit
codes reporting 0 for failures, and a prune reconciliation path that could
erase an entire store. Verified end to end against a real Aurora
PostgreSQL 16.11 cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jdurham38
jdurham38 requested a lite review from Copilot August 7, 2026 22:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@jdurham38
jdurham38 requested review from iamh2o and a balanced review from Copilot August 7, 2026 22:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Formalize the TapDB backup and recovery lifecycle across CLI, API, and GUI

2 participants