Skip to content

runbook.md

Latest commit

 

History

History
36 lines (25 loc) · 1.53 KB

runbook.md

File metadata and controls

36 lines (25 loc) · 1.53 KB
Raw

Operations runbook

Local recovery

  1. Check GET /health and inspect the latest row in pipeline_runs.
  2. Run c360 process; pending Bronze events are safe to replay.
  3. Run c360 validate to check identity, SCD, and profile invariants.
  4. For a disposable local reset, run make seed. Do not use this reset on persistent environments.

Connector failure

  1. Record connector/task state and MilliSecondsBehindSource before restarting.
  2. Check PostgreSQL replication-slot retained bytes or MySQL binlog availability.
  3. Restart only the failed task. Do not delete a slot or connector offset unless a controlled resnapshot has been approved.
  4. Confirm offsets advance, DLQ remains empty, and source/Bronze reconciliation returns to zero.

Identity super-node

  1. Stop publishing the affected xref run to Gold.
  2. Inspect identity_quarantine and the high-degree identifier.
  3. Add a reviewed block/guard rule; never manually edit only the Gold profile.
  4. Replay identity edges, compare precision/recall, publish remaps, then rebuild affected profiles.

Backfill

Pin the input Bronze snapshot/offset range and identity-model version. Write to an Iceberg branch, run reconciliation and no-future-leakage checks, then fast-forward after approval. Backfills must not overwrite raw Bronze data.

Privacy erasure

Resolve all identifiers to the master ID, tombstone source records, purge Redis keys, delete affected Iceberg rows, expire snapshots after the legal retention hold, and write an immutable audit event containing no erased PII.