feat(backfill): move OwnerBackfill drain to a standalone worker (COW-1118)

[yvesfracari]

Closes COW-1118.

Problem

The non-deterministic order drain (COW-1117) ran inside a Ponder block handler (OwnerBackfill:block / OwnerBackfillLive:block). Ponder executes indexing functions serially in checkpoint order, so:

1. Inline execution — a 30s /account call parked event indexing for 30s; the "overlaps historical sync" claim was false.
2. Non-atomic, non-resumable commit — fetchComposableOrders fetched all pages then committed once. A timeout mid-pagination abandoned the promise before any write, so the next firing restarted from the same cursor → timed out again → historyBackfilled never flipped → /readyz stayed 503 forever. Any owner too big to drain in 30s wedged readiness permanently (this is why prod was stuck).

Fix

Move the HTTP drain into a separate long-running worker that coordinates with the indexer only through the durable cow_cache schema. Goal: time-to-ready ≈ max(backfill, drain) instead of backfill + drain, with no owner able to wedge readiness.

Changes

• New table cow_cache.owner_drain_state (chain_id, owner, status, next_offset, claimed_at, updated_at) — durable work queue. CREATE TABLE IF NOT EXISTS via createCowCacheTables, called by both setup.ts and the worker (not startup-ordering-dependent).
• Two ponder-free modules so the worker can share the decode/hash/cache logic without importing ponder:schema:
  • orderbookHttp.ts — HTTP client (retry/backoff, pagination, new fetchAccountOrderPage for the offset-walk).
  • composableCache.ts — cow_cache tables, filterAndProcessForCache (decode → reconstruct generator_hash, no DB), cache upsert/read, claim/complete/release helpers.
• Worker (src/worker/drain.ts, pnpm drain) — claims owners with FOR UPDATE SKIP LOCKED + stale-lease reclaim, drains ~10 concurrently, offset-walks newest→oldest committing each page and bumping next_offset, marks complete at a short page. Touches only cow_cache, so it's deployment-independent and safe to run N-wide.
• Ponder side (HTTP-free): ConditionalOrderCreated enqueues non-deterministic owners (ON CONFLICT DO NOTHING); OwnerBackfill:block is now a DB-only projection of complete owners (hash → current eventId, orphan-drop, flip historyBackfilled). OwnerBackfillLive removed; OwnerBackfill is one registration with no endBlock. The redundant reconcile step was dropped (OrderStatusTracker already polls open orders).
• Deployment: drain sidecar in docker-compose.yml; pg dependency; docs updated (deployment.md, architecture.md, agent_docs/, .env.example).

Verification

• pnpm codegen / typecheck / lint clean; 135/135 tests pass (obsolete fetchComposableOrders tests replaced with fetchAccountOrderPage + filterAndProcessForCache coverage).
• Drove the worker against a local Postgres: bootstraps both tables, idle-sleeps on empty queue, claims pending and reclaims a stale draining lease (preserving next_offset), marks owners complete, shuts down cleanly on SIGTERM.
• Confirmed the worker's module graph resolves outside the Ponder runtime, and that pnpm v10 keeps tsx in the NODE_ENV=production image so the sidecar runs.

Reviewer note — deliberate trade-off

The projection runs at interval: 250 (a single registration can't have different backfill vs. realtime cadences), so a post-tip straggler adds up to one interval (~20 min gnosis / ~50 min mainnet) to time-to-ready. Bounded and can never wedge — matches the issue's "remove OwnerBackfillLive" instruction. If snappier post-tip readiness is preferred, this can be split into two projection-only registrations (coarse backfill + fine live).

[linear-code]

COW-1118