Skip to content

round: arm the status-reconcile clock for every checkpointed round - #1052

Open
Roasbeef wants to merge 9 commits into
mainfrom
boarding-checkpoint-liveness
Open

round: arm the status-reconcile clock for every checkpointed round#1052
Roasbeef wants to merge 9 commits into
mainfrom
boarding-checkpoint-liveness

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Jul 27, 2026

Copy link
Copy Markdown
Member

A boarding deposit is real money the user parked in an on-chain address to
convert into an off-chain VTXO, and partway through that conversion the
client hits a point of no return where it has signed and can only wait. If
the operator drops the round at exactly that moment, say its finalize write
fails so it rolls back before broadcasting, then nothing will ever tell the
client: no transaction exists to confirm, and no round record survives to
send a failure. The client had one remaining way to find out, a timer that
asks the operator "is this round dead?", but that timer was only armed for
rounds carrying forfeits, which a boarding-only round has none of. So the
client waits forever, the user sees a deposit that never became spendable
and no error explaining why, and the only way to get the coins back is to
wait out the CSV timelock and exit unilaterally. This PR arms that timer for
every round, so the client asks, gets told the round is dead, and fails
cleanly instead of hanging, and gives the deposit back.

In FSM terms, InputSigSentState has exactly three exits: the commitment
confirms, the operator delivers a failure, or the reconcile probe returns an
authoritative status. The probe's timer arrived with #844, whose hazard is
releasing forfeit reservations before the round's fate is known, so every
site that touched it gated on a non-empty forfeit set. Read on its own that
gate looks like an optimization, since a boarding-only round has no
reservations to reconcile. What it actually does is shut all three exits at
once. Widening it needs no new protocol surface and no new state, since a
dead status already fails the round and releaseForfeitsOnFailure over an
empty forfeit set is a no-op.

So we arm the clock on all three doors into the state: the
forfeit-collection transition, the PartialSigsSentState transition a
boarding-only round takes instead (the door #1051 actually walks through),
and the recoverActiveRounds reload on restart. All four exits disarm, and
the timeout handler probes rather than self-looping when the round carries
no forfeits. That leaves a simpler invariant than the one it replaces: the
clock is armed for the whole of InputSigSentState, and every exit disarms
it.

Ordering is part of that contract. processOutbox abandons the rest of the
outbox on the first failing Tell, and the FSM has already checkpointed by
dispatch time, so the arm leads the fallible sends; arming last would reopen
this same strand through a different door. The disarms trail every delivery
for the mirror-image reason, since cleanup must never gate confirmed funds
or a terminal job drop.

How it was found, and how it is verified

The phase 5 DST universal quiescence oracle, not a report. At the end of
every workload run the harness fast-forwards past every TTL, lets the
reconcilers fire, then asserts that nothing money-bearing is parked in a
state something was supposed to move. [materialize, refreshOk, storeFault]
passes every per-op assertion in the suite and is caught only by that
end-of-run check:

client round 00dc6ad0-2900-7035-bfbd-5ff7ccbfd3eb parked in non-terminal
state *round.InputSigSentState at quiescence: its round is dead on both
sides and nothing will ever move it again

We pointed the harness at this branch with the suppressing classification
deleted, making the oracle unconditional: full suite green (90.9s), 500-seed
soak green. In-repo, checkpoint_arming_test.go and
status_reconcile_test.go cover every door and exit and assert the two
orderings by position rather than presence; every case fails against the
pre-fix code.

Giving the deposit back

Un-parking the FSM is not the same as getting the money back, so the second
half of this PR does that. RoundStore had no counterpart to
FinalizeRound, so a failed round kept its checkpoint row: it stayed in
ListActiveRounds and was re-hydrated on every start, and the boarding
intents it adopted stayed adopted. Since boardingIntentSweepable excludes
adopted, the deposit was neither boardable nor sweepable, not before the
CSV and not after it either, because nothing writes the expired status. The
coins stayed pinned against the board limit with no way to reach them.

The recovery machinery turned out to be already built and simply
unreachable. ListBoardingIntentsByStatus and
ListBoardingIntentsBySweepableStatuses both re-admit a confirmed intent
exactly when its linked round reads failed, and round_statuses has
carried a failed row since the schema landed, commented "Round failed,
intents may need recovery". Nothing ever wrote it.

So FailRound writes it, and reverts the round's adopted intents in the
same transaction. Intents revert to confirmed rather than failed, since
a dead round proves the commitment never broadcast: nothing on-chain failed,
and the UTXO is exactly as it was before the round. That restores both
recovery routes at once, and the revert is guarded on adopted in SQL so a
sweep already in flight is never clobbered. The actor calls it from the
RoundFailedNotification handler, the one choke point every failure path
already passes through.

Fixes #1051

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60e9fc5be3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread round/transitions.go
Comment on lines +4430 to 4431
if env.StatusReconcileTimeout <= 0 {
return selfLoop(s), nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Persist dead boarding rounds before leaving reconciliation

When this newly enabled boarding-only timeout receives a ROUND_STATUS_DEAD response, the existing branch only changes the in-memory FSM to ClientFailedState; the checkpoint row remains input_sig_sent and its boarding intents remain BoardingStatusAdopted. Consequently ListActiveRounds reloads the round after every restart, while the boarding sweep actor explicitly excludes adopted intents, so the deposit still cannot enter CSV recovery. The dead path needs to retire the persisted checkpoint and mark its boarding intents failed, not merely stop the in-memory reconciliation loop.

AGENTS.md reference: round/AGENTS.md:L167-L171

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codex You are right on the facts, and this is the most important finding on the PR, but I am deliberately not fixing it here.

Verified both halves independently:

  • RoundStore exposes only ListActiveRounds and FinalizeRound (round/interfaces.go), with no fail or delete path, so a round entering ClientFailedState leaves its checkpoint row active and gets re-hydrated on every restart. reapFailedRounds only drops the in-memory FSM.
  • Nothing in production ever writes BoardingStatusFailed or BoardingStatusExpired — grep turns up only the enum definitions and the sweep filter — and boardingIntentSweepable returns false for Adopted (wallet/boarding_sweep_actor.go), so the deposit does not enter early sweep recovery.

So the honest scope of this PR is narrower than its body claimed: it un-parks the FSM, but the money still does not come back before CSV. I have said so in the PR description.

Why it is not a fixup: the delivered-failure path has had this identical shape since well before this change, so it is inherited, not introduced. Closing it needs a new outbox message plus wallet/DB plumbing to revert Adopted and retire the persisted checkpoint — a separate change with its own tests, not something to bury in a fixup on this one. Flagging for a follow-up rather than filing unilaterally.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread round/transitions.go
Comment thread round/transitions.go Outdated
@litbot-9000

litbot-9000 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

This PR's Go changes touch one documented package — round/ — and the
CLAUDE.md/AGENTS.md pair it updates names a symbol that does not exist:
the restart re-arm of the status-reconcile clock lives in the
ListActiveRounds resume loop in RoundClientActor.Start
(round/actor.go:1373-1437), not in a recoverActiveRounds function.

diff --git a/round/AGENTS.md b/round/AGENTS.md
index 6c0997e3..24377625 100644
--- a/round/AGENTS.md
+++ b/round/AGENTS.md
@@ -185,12 +185,13 @@ state transitions and validation rules live under [Invariants](#invariants).
   exit disarms it. There are three doors: `forfeitCollectionOutbox`
   (forfeit-bearing rounds), the `PartialSigsSentState` →
   `InputSigSentState` transition (boarding-only rounds, which never enter
-  forfeit collection), and `recoverActiveRounds` on restart. Arming is
-  **not** gated on `len(Intents.Forfeits) > 0`: for a forfeit-bearing round
-  the probe gates the reservation release on an authoritative dead answer,
-  but for *any* round it is the sole liveness clock in the checkpointed
-  state — an operator that rolls the round back before broadcast produces
-  no confirmation and no failure, so an unarmed boarding-only round strands
+  forfeit collection), and the `ListActiveRounds` resume loop in
+  `RoundClientActor.Start` on restart. Arming is **not** gated on
+  `len(Intents.Forfeits) > 0`: for a forfeit-bearing round the probe gates
+  the reservation release on an authoritative dead answer, but for *any*
+  round it is the sole liveness clock in the checkpointed state — an
+  operator that rolls the round back before broadcast produces no
+  confirmation and no failure, so an unarmed boarding-only round strands
   its deposit until the CSV expires.
 - **Reconcile outbox ordering.** `processOutbox` abandons the rest of the
   outbox on the first failing `Tell`, and the FSM has already checkpointed
diff --git a/round/CLAUDE.md b/round/CLAUDE.md
index 6c0997e3..24377625 100644
--- a/round/CLAUDE.md
+++ b/round/CLAUDE.md
@@ -185,12 +185,13 @@ state transitions and validation rules live under [Invariants](#invariants).
   exit disarms it. There are three doors: `forfeitCollectionOutbox`
   (forfeit-bearing rounds), the `PartialSigsSentState` →
   `InputSigSentState` transition (boarding-only rounds, which never enter
-  forfeit collection), and `recoverActiveRounds` on restart. Arming is
-  **not** gated on `len(Intents.Forfeits) > 0`: for a forfeit-bearing round
-  the probe gates the reservation release on an authoritative dead answer,
-  but for *any* round it is the sole liveness clock in the checkpointed
-  state — an operator that rolls the round back before broadcast produces
-  no confirmation and no failure, so an unarmed boarding-only round strands
+  forfeit collection), and the `ListActiveRounds` resume loop in
+  `RoundClientActor.Start` on restart. Arming is **not** gated on
+  `len(Intents.Forfeits) > 0`: for a forfeit-bearing round the probe gates
+  the reservation release on an authoritative dead answer, but for *any*
+  round it is the sole liveness clock in the checkpointed state — an
+  operator that rolls the round back before broadcast produces no
+  confirmation and no failure, so an unarmed boarding-only round strands
   its deposit until the CSV expires.
 - **Reconcile outbox ordering.** `processOutbox` abandons the rest of the
   outbox on the first failing `Tell`, and the FSM has already checkpointed

The rest of the reconcile documentation this PR added checks out against the
code: both in-FSM arm sites (round/transitions.go:2919, :3398), the four
disarm sites covering every exit from InputSigSentState, the lead/trail
outbox ordering, and statusReconcileMaxBackoffShift /
defaultStatusReconcileTimeout (90 s) all match.

How to apply: save the diff above and git apply it, or run the
doc-gardening skill locally over round/.

make doc-check reports 10 pre-existing errors in other packages
(.claude-pr, db, ledger, lib/types, oor, sdk/ark,
sdk/wavewalletdk, swapwallet, vtxo, waved). They predate this PR and
are out of scope for this advisory; the round/ pair is clean.

https://github.com/lightninglabs/wavelength/actions?query=branch%3Aboarding-checkpoint-liveness

@Roasbeef Roasbeef added the backport-v0.1.x-branch Backport this merged PR to v0.1.x-branch label Aug 6, 2026
@Roasbeef

Roasbeef commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

@claude review this

@claude

claude Bot commented Aug 7, 2026

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 5m 42s —— View job


Review: round: arm the status-reconcile clock for every checkpointed round

I traced the four touched sites (forfeitCollectionOutbox, the two InputSigSentState failure exits, the confirmation exit, and the recoverActiveRounds re-arm) plus processOutbox and the actor's failure/complete notification handling. The core change is sound and the invariant it establishes — the clock is armed for the whole of InputSigSentState and every exit disarms it — is genuinely simpler than the forfeit-count gate it replaces. Below is my take on the three Codex findings plus what I independently corroborated.


1. Codex P1 — "persist dead boarding rounds before leaving reconciliation" — worth confirming; I partially corroborate it

This is the finding that matters. Tracing the failure path for a boarding-only round:

  • The dead-status handler (round/transitions.go:4527) and the delivered-failure shortcut (round/transitions.go:4374) both produce a ClientFailedState transition only. releaseForfeitsOnFailure over an empty forfeit set is a no-op, as advertised.
  • On the actor side, RoundFailedNotification (round/actor.go:2740) is observability-only — it logs and bumps a counter. No FinalizeRound, no CommitState rewrite, no FailForfeitIntents (empty set). Only TerminalJobFailedNotification drops durable job state, and that rides on a non-empty forfeit release.

So after the probe fails a boarding-only round:

  • The durable checkpoint row stays input_sig_sent, so ListActiveRounds re-hydrates it on the next start, recoverActiveRounds re-arms, the probe fires, gets DEAD again, and re-fails in memory. The failure is not durable — every restart resurrects the round into InputSigSentState transiently and re-probes. Churny, self-correcting, but not the clean terminal state the commit message implies.
  • The boarding intents stay BoardingStatusAdopted. wallet/boarding_sweep_actor.go:580 (boardingIntentSweepable) returns false for Adopted, and ListBoardingIntentsBySweepableStatuses only covers Confirmed/Failed/Expired. I could not find anything on the round-FSM failure path that reverts Adopted → Failed/Expired, so the deposit does not enter early sweep recovery — it stays counted as adopted balance (wallet/board_limits.go:348, wallet/wallet.go:1010).

The honest framing: this is the same in-memory-only behavior the pre-existing delivered-failure path already had for boarding-only rounds — the PR doesn't introduce it, it inherits it onto a second entrance. So either (a) there's an out-of-band reversion of Adopted intents I didn't find (e.g. via pending-intent replay on Recoverable: true), in which case both paths are fine; or (b) it's a real shared gap. The reason I'd ask you to confirm rather than dismiss: the PR's stated goal is "the deposit converts or a failure surfaces and the money comes back" — but if the intent stays Adopted, the FSM un-parks yet the deposit still only recovers at CSV, which is the very outcome the PR sets out to avoid. The DST quiescence oracle caught the FSM parking; if it doesn't model boarding-intent DB status, a stuck-Adopted deposit would pass it. Worth a direct check: does anything move a boarding intent out of Adopted when its round FSM fails?

2. Codex P1 — "arm the reconcile timeout before the fallible sends" — valid, cleanest to act on

In forfeitCollectionOutbox (round/transitions.go:2921) the StartTimeoutReq is appended after SubmitVTXOForfeitSigsToServer, the boarding-sig submission, and RegisterConfirmationRequest. processOutbox (round/actor.go:2506) returns on the first failing Tell, and the FSM is already checkpointed into InputSigSentState before the outbox runs. So a failed earlier send leaves a checkpointed round with no armed timer for the rest of the session — exactly the strand this PR is closing, just triggered by a send error instead of an operator rollback. The new recoverActiveRounds re-arm covers it after a restart, so it's not permanent, but arming the timer first would make the "armed for the whole of InputSigSentState" invariant hold within the session too, and arming-first is strictly safer for liveness (a subsequent send error just means the timer fires and probes). Note Codex's phrasing "as the forfeit-collection transitions already do" is slightly off — this is that transition, and it currently arms last.

Fix this →

3. Codex P2 — "cancel the timer after / independently of the terminal notifications" — valid, low severity

On the confirmation exit (round/transitions.go:4629) cancelStatusReconcileTimeout is now the first outbox entry. If the timeout actor rejects that Tell, processOutbox aborts before VTXOCreatedNotification and RoundCompletedNotification, so onRoundComplete/FinalizeRound never runs even though the VTXOs are built and the FSM moved to ConfirmedState — a stale-timer cancel would block confirmed-funds delivery. A CancelTimeoutReq Tell only fails if the timeout actor is down/saturated, so this is unlikely, but making the cancel best-effort (mirroring the ReleaseForfeitReservation fire-and-forget branch already in processOutbox) or ordering it last is cheap and removes the coupling. The same prepend-then-fallible-release shape also appears on the dead-status exit at round/transitions.go:4527.


Summary

  • The widening is correct and the resulting invariant is cleaner; both exits disarming the timer is a real improvement.
  • multI: add initial repo scaffolding #2 is the highest-value follow-up and directly strengthens the invariant this PR is selling — I'd take it.
  • lib: start adding lib helpers #1 is the one I'd want you to confirm out loud: is there a mechanism that reverts Adopted boarding intents (and/or retires the checkpoint row) when the FSM fails? If not, the fix un-parks the FSM but may leave the deposit recovering only at CSV — and the delivered-failure path shares that gap, so it's not new, just newly reachable.
  • chainbackend+chainsource: add chainsource actor and backend impl #3 is a cheap robustness nit — make the cancel best-effort.

None of these are blockers for the parking fix itself; #1 is the one worth a definitive answer before this rides the v0.1.x backport.
· boarding-checkpoint-liveness

@Roasbeef
Roasbeef force-pushed the boarding-checkpoint-liveness branch from 60e9fc5 to c4986ec Compare August 7, 2026 23:47
@Roasbeef

Roasbeef commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

Pushed a squashed update covering the review round (Codex, @claude, plus a Fable adversarial pass). Dispositions below.

Fixed

Codex P1 — arm the reconcile timeout before the fallible sends. Confirmed and fixed. processOutbox returns on the first failing Tell, and the FSM is already checkpointed into InputSigSentState by the time the outbox is dispatched, so arming last let a mid-flight send error reopen this very strand through a different door. The arm now leads the server sends. Rebuilding that outbox also let the outboxMsgs[:2]/outboxMsgs[2:] splice go, which is a readability win on its own.

Codex P2 — cancel after the terminal notifications on the confirmation path. Confirmed and fixed, and it pairs with the above as one rule: the arm leads the fallible sends, the disarm trails them. Cleanup must never gate delivery. A cancel that never lands only leaks a one-shot timer that fires into a terminal state and self-loops; a cancel that runs first could withhold already-persisted VTXOs from the manager and leave onRoundComplete unfinalized.

Fable — the buildClientVTXOs error exit did not disarm. That branch also leaves InputSigSentState, so the "every exit disarms" claim was true of most branches rather than all of them. It disarms now, via a shared reconcileDisarmEvents helper that also replaces the fn.None/fn.Some dance in the delivered-failure branch and the inline construction on the dead-answer path.

Fable — two stale comments in fsm_timeouts.go. TimeoutPhaseStatusReconcile still said "armed when the forfeit signatures leave the box", and cancelStatusReconcileTimeout still listed only two callers. Both corrected; a stale comment is worse than none.

Tests. The PR previously carried none, which was the fair criticism. Five cases in status_reconcile_test.go: a boarding-only round probes and re-arms on expiry, disarms on a delivered failure, and emits no cancel when the reconcile is opted out; the two orderings above are asserted by position, not presence. Verified the tests fail without the code: the two boarding cases fail against a7f12c11 (pre-PR), and the two ordering cases fail against 60e9fc5b (pre-fixup).

Not fixed here — real, but pre-existing and wider than this PR

Codex P1 — dead rounds are not retired durably, and boarding intents stay Adopted. This one is correct and it is the most important thing on the PR, but it is not this PR's regression. Tracing it: RoundStore exposes only ListActiveRounds and FinalizeRound, with no fail/delete path, so a round entering ClientFailedState leaves its checkpoint row active and gets re-hydrated on every restart. Separately, nothing in production ever writes BoardingStatusFailed or BoardingStatusExpired — grep turns up only the enum definitions and the sweep filter — and boardingIntentSweepable returns false for Adopted, so the deposit does not enter early sweep recovery.

The consequence is that this PR un-parks the FSM but does not yet get the money back before CSV, which is narrower than the PR body claims. The delivered-failure path has had this same shape since before this change, so fixing it means a new outbox message plus wallet/DB plumbing to revert Adopted and retire the checkpoint — a separate change, not a fixup. Flagging rather than filing so we can decide where it belongs.

Fable — the post-admission span (QuoteReceivedPartialSigsSent) has no liveness clock, since RoundJoined cancels the registration watermark and quote expiry is only checked when a quote actually arrives. Same shape as this bug, softer blast radius: those states are in-memory, so a restart drops the round and replays the pending intent. Also pre-existing and out of scope.

go test ./round/... green, make lint-changed-local reports 0 issues, make commitmsg-lint OK.

@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

@claude I pushed a squashed update addressing your review and the Codex findings. Please re-review c4986ece.

Actioned: the reconcile arm now leads the fallible sends in forfeitCollectionOutbox; the confirmation-path disarm now trails the terminal notifications rather than leading them; the buildClientVTXOs error exit disarms too (it left InputSigSentState without cancelling, so the "every exit disarms" claim held on most branches rather than all); the two stale comments in fsm_timeouts.go are corrected; and there are now five regression cases in status_reconcile_test.go.

On your question — "does anything move a boarding intent out of Adopted when its round FSM fails?" — I checked, and the answer is no. Nothing in production writes BoardingStatusFailed or BoardingStatusExpired, and boardingIntentSweepable returns false for Adopted. So it is your option (b), a real shared gap, not an out-of-band reversion I missed. It is pre-existing rather than introduced here, so I documented it in the PR body under "What this does not close" instead of expanding scope.

Two things worth your attention on the re-review: whether the arm-leads/disarm-trails ordering rule is now applied consistently across every site in this state, and whether the new tests actually pin the behavior rather than restating the implementation.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 3m 54s —— View job


Re-review of boarding-checkpoint-liveness (squashed update + 43c2adf)

I traced every entry and exit of InputSigSentState against the two questions you flagged. Both the ordering rule and the tests hold up well; I found one narrow ordering asymmetry and two coverage observations worth your call. Nothing here is a blocker.

Note: the branch head is now 43c2adf (the boarding-only-door commit) on top of c4986ec, so this review covers both.


Q1 — is arm-leads / disarm-trails applied consistently across every site? Almost — one exception on the dead-answer path.

Entry doors (all arm before any fallible send, all gated on StatusReconcileTimeout > 0):

  • forfeitCollectionOutbox — arm now precedes SubmitVTXOForfeitSigsToServer, SubmitForfeitSigRequest, RegisterConfirmationRequest. ✓ (and the outboxMsgs[:2]/[2:] splice is gone — nice)
  • PartialSigsSentState boarding-only branch (transitions.go:3381) — arm precedes forfeitSigReq + RegisterConfirmationRequest. ✓
  • recoverActiveRounds restart re-arm (actor.go:1416) — standalone processOutbox([StartTimeoutReq]), nothing fallible trails it. ✓

Exit doors:

  • BoardingFailed boarding-only (4402) → reconcileDisarmEvents, outbox is just the cancel. ✓
  • BoardingConfirmed build error (4578) → reconcileDisarmEvents. ✓
  • BoardingConfirmed success (4691) → cancel trails both notifications. ✓ (Codex P2 fixed)
  • RoundStatusReported dead (4540) → reconcileDisarmEvents, then wrapped by releaseForfeitsOnFailure. ⚠️ see below

The exception. On the dead-answer exit, reconcileDisarmEvents seeds Outbox = [cancel], then releaseForfeitsOnFailure prepends the rollback and appends the job-drop notification (transitions.go:178 + :195). The final order is:

[ReleaseForfeitReservation…, cancel, TerminalJobFailedNotification]

ReleaseForfeitReservation is fire-and-forget in processOutbox (actor.go:2571), so the #844-critical release is never gated by the cancel — good. But CancelTimeoutReq does return an error and abort the loop (actor.go:2560-2569), and it sits before TerminalJobFailedNotification. So on a terminal-for-job dead answer, a rejected cancel (timeout actor down/saturated) short-circuits before the job-drop fires, leaving the pending intent in recoverable replay — the exact outcome that block at transitions.go:181-203 exists to prevent. This is the same shape as Codex P2, just on the dead-answer door rather than the confirmation door, so the "disarm trails all delivery" rule doesn't strictly hold here.

Severity is low: the window is narrow, the release (the money-relevant delivery) already ran, and the consequence is a recoverable replay rather than lost funds. But since the PR sells a uniform invariant, it's the one spot where cleanup can still gate a delivery. Cheapest fixes, either: append the cancel after releaseForfeitsOnFailure returns (so it's genuinely last), or make CancelTimeoutReq best-effort in processOutbox (then the whole ordering question dissolves for every phase). Fix this →

Otherwise the invariant holds: every state-changing exit disarms, every self-loop (StatusReconcileTimedOut re-probe, non-dead / not-ours RoundStatusReported, default) re-arms via statusReconcileProbeOutbox or stays put. Good.


Q2 — do the tests pin behavior or restate implementation? They pin behavior.

  • TestReconcileArmedBeforeFallibleSends / assertArmsReconcileFirst — assert the arm index is strictly below each fallible send by position, with an explicit carve-out only for a bookkeeping CancelTimeoutReq. This is a real ordering contract: it fails if the arm moves back behind a send, which is exactly the regression you're guarding. Not a restatement.
  • TestConfirmationDisarmTrailsNotifications — asserts doneIdx < cancelIdx. Mirror-image, behavioral.
  • TestBoardingOnlyReconcileTimeoutProbes — asserts the boarding-only expiry probes + re-arms rather than self-looping; fails against the old forfeit-count gate.
  • TestBoardingOnlyDeliveredFailureDisarms + TestDeliveredFailureNoDisarmWhenReconcileDisabled — pin both sides of the arm-gate (cancel present when armed, absent when opted out).
  • TestBoardingDoorArmsReconcileClock drives a real OperatorSigned through the FSM and asserts it lands in InputSigSentState before checking the arm (checkpoint_arming_test.go:215), so it can't pass over a round that never reached the hazard window. The "harness leaves the timeout at zero" note is the honest explanation for why this went unseen — good.

Two coverage observations (not blockers):

  1. The restart-reload door is unpinned. checkpoint_arming_test.go:46 names "two live doors plus the restart reload," but only the two live doors get tests. Per 43c2adf's own message the restart path was the only thing recovering boarding-only rounds before this fix — i.e. the highest-value door — yet the recoverActiveRounds re-arm has no regression test. A direct test that reloads a checkpointed boarding-only round and asserts the StartTimeoutReq goes out would close the last gap in "every door arms."
  2. The dead-answer exit ordering is unpinned. Consistent with Q1: there's no positional assertion on the RoundStatusReported dead path, which is why the cancel-before-TerminalJobFailedNotification ordering slipped through. If you act on Q1, a positional test there would lock it.

Verdict

The widening is correct, the invariant is cleaner than the forfeit-count gate it replaces, and Codex P1(order)/P2 plus the Fable findings are all genuinely addressed. On your two questions: the ordering rule is consistent at every site except the dead-answer terminal-for-job path, where the disarm still leads one delivery (low severity, recoverable, same shape as the P2 you already fixed); and the tests pin behavior by position rather than restating the implementation, with the restart-reload door being the one door left without a regression test.

On the Adopted-intent gap: agreed it's pre-existing and correctly scoped out — the PR body's "What this does not close" now states it plainly, so the description no longer oversells. That's the right call for the v0.1.x backport.

I couldn't run go test/lint in this environment (both need approval), so the test results above are from static tracing plus your reported green run — I did not independently execute them.
· boarding-checkpoint-liveness

In this commit, we widen the status-reconcile timer so it covers
boarding-only rounds, not just the forfeit-bearing ones. The timer was
introduced for wavelength#844, where the hazard is releasing forfeit
reservations before the round's fate is known, so every site gated
itself on a non-empty forfeit set. That gate reads as an optimization,
but it quietly leaves boarding-only rounds with no liveness clock at
all.

Once a round reaches InputSigSentState its only exits are a commitment
confirmation, a delivered failure, or the reconcile probe. When the
operator rolls the round back before broadcast, no commitment can ever
confirm and there is no durable round left to fail the client with, so
a boarding-only round parks in InputSigSentState forever: the deposit
never converts, nothing surfaces to the user, and only the CSV timeout
gets the funds back. The probe already handles this exact answer, since
a dead status fails the round and a release over an empty forfeit set is
a no-op.

We arm the clock on both doors into the state, the live signing
transition and the restart path that reloads a checkpointed round, and
teach the timeout handler to probe rather than self-loop when the round
carries no forfeits. The two exits that previously left the timer
running for these rounds now cancel it: the delivered-failure shortcut
and the confirmation path, which both drop their forfeit-count gate.
That leaves a simpler invariant than the one it replaces, since the
clock is armed for the whole of InputSigSentState and every exit
disarms it.

Because that clock is now the only thing keeping a checkpointed round
from parking forever, the ordering around it matters. processOutbox
abandons the rest of the outbox on the first failing Tell, so the arm
moves ahead of the fallible server sends: arming last would let a
mid-flight send error reopen the same strand through a different door.
The disarms move the other way, behind every delivery: on the
confirmation path behind the VTXO and round-completion notifications,
and on the dead-answer path behind the forfeit release and the job
drop. A cancel is one of the few outbox entries that can fail, so
leading with one lets a saturated timeout actor withhold
already-persisted funds from the manager, or strand a pending intent in
recoverable replay that the job drop exists to retire. Cleanup must
never gate delivery. The confirmation path's VTXO-build error exit
disarms as well, so the every-exit half of the invariant holds on every
branch rather than most of them.

The new cases in status_reconcile_test.go pin both halves: a
boarding-only round probes and re-arms on expiry and disarms on a
delivered failure, and the two orderings above are asserted by
position rather than by presence.

Fixes #1051
@Roasbeef
Roasbeef force-pushed the boarding-checkpoint-liveness branch from 43c2adf to e9535bd Compare August 8, 2026 00:36
@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

@claude Addressed the dead-answer ordering exception you found in the re-review. Verified it first: releaseForfeitsOnFailure prepends the rollback and appends the job drop, so seeding the cancel into the transition put it at index 1 of [Release…, cancel, TerminalJobFailedNotification], and since ReleaseForfeitReservation is fire-and-forget while CancelTimeoutReq returns an error, a rejected cancel really could suppress the job drop and leave the pending intent in recoverable replay.

Took the local fix rather than making CancelTimeoutReq best-effort globally. The transition no longer seeds the disarm; instead a new appendReconcileDisarm adds it after releaseForfeitsOnFailure returns, so the cancel is genuinely last on that door too. Making the cancel best-effort in processOutbox is the more general answer and would dissolve the ordering question for every phase, but it changes semantics for the forfeit-collection and registration phases as well, which is wider than this PR should reach — noting it rather than doing it.

TestDeadStatusTerminalCodeRetiresJob now asserts notifyIdx < cancelIdx; it fails on the previous code with "2 is not less than 1".

So the rule is now uniform across all four exits: the arm leads the fallible sends, the disarm trails every delivery.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 2m 27s —— View job


Re-review of the dead-answer ordering fix

Traced the fix for the exception I flagged in the last round. It's correct and it closes the last asymmetry cleanly.

The ordering now holds on the dead-answer door. The transition seeded into releaseForfeitsOnFailure at transitions.go:4540 no longer carries the cancel. The wrapper lays down [ReleaseForfeitReservation…, TerminalJobFailedNotification] (prepend at :178, append at :195), and the new appendReconcileDisarm (fsm_timeouts.go:121) tacks the cancel on after the wrapper returns (transitions.go:4569). Final order for a terminal-for-job dead answer:

[ReleaseForfeitReservation…, TerminalJobFailedNotification, CancelTimeoutReq]

So the cancel is genuinely last. ReleaseForfeitReservation stays fire-and-forget and the job drop now sits ahead of the one outbox entry that can abort the loop — a rejected CancelTimeoutReq can no longer suppress the job retirement, which was the recoverable-replay strand. The empty-forfeit boarding-only sub-case is also fine: releaseForfeitsOnFailure returns the transition untouched (:152), then appendReconcileDisarm yields [CancelTimeoutReq] — present and last. The gate (StatusReconcileTimeout <= 0 → no-op) mirrors the arm site, so no cancel is emitted for a timer that was never scheduled.

The rule is now uniform across all four exits (arm leads fallible sends, disarm trails every delivery):

Exit Site Disarm placement
BoardingFailed boarding-only 4402 reconcileDisarmEvents — cancel is the sole entry ✓
RoundStatusReported dead 4569 appendReconcileDisarm, after release + job drop ✓ (the fix)
BoardingConfirmed build error 4598 reconcileDisarmEvents — cancel is the sole entry ✓
BoardingConfirmed success 4706 appended after all VTXO/completion/forfeit-confirmed notifications ✓

The forfeit-bearing BoardingFailed branch parks and re-probes rather than exiting, so it correctly does not disarm. And 4569 is the only InputSigSentState caller of releaseForfeitsOnFailure, so there's no other site where a seeded cancel could sneak mid-outbox.

Choosing the local fix over global best-effort is the right call for this PR. Making CancelTimeoutReq best-effort in processOutbox would dissolve the ordering question everywhere, but it changes the abort semantics for the forfeit-collection and registration phases too — wider than a boarding-liveness PR should reach. appendReconcileDisarm keeps the change on exactly the door that needed it, and the helper comment records why trailing is mandatory.

The test pins behavior by position. TestDeadStatusTerminalCodeRetiresJob (status_reconcile_test.go:396) asserts notifyIdx < cancelIdx via outboxIndexOf, not mere presence — under the pre-fix order ([Release…, cancel, notify]) it's require.Less(2, 1), which fails exactly as you reported ("2 is not less than 1"). It also checks the notification carries the right ForfeitOutpoints/FailureCode, so it's a real terminal-for-job composition test, not a restatement.

Verdict

The dead-answer exception is resolved, and the arm-leads / disarm-trails invariant is now genuinely uniform across every InputSigSentState exit. No further findings on this path. The two coverage gaps I noted last round (a direct regression test for the recoverActiveRounds restart-reload door, and — now closed — the dead-answer ordering) are down to just the restart-reload door, which remains the one door in "every door arms" without its own test; still not a blocker. The Adopted-intent gap stays correctly scoped out in the PR body. This looks ready for the v0.1.x backport.

I couldn't run go test/lint here (both need approval), so the pass/fail claims above are from static tracing plus your reported green run — I did not independently execute them.
· boarding-checkpoint-liveness

@Roasbeef
Roasbeef force-pushed the boarding-checkpoint-liveness branch from e9535bd to d70053e Compare August 8, 2026 01:16
@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

Closed the last coverage gap from the re-review: TestRestartDoorArmsReconcileClock in checkpoint_arming_test.go now pins the restart-reload door, which was the one door in "every door arms" without its own test.

It reloads a boarding-only checkpointed round through Start and asserts the status-reconcile timeout is actually scheduled. It fails against a7f12c11 with the old forfeit-count gate, which is the case that mattered doubly: while the live boarding door was unarmed, the restart path was the only thing that could still rescue such a round, and it was gated on the same count.

All three doors and all four exits now have direct coverage.

In this commit, we close the last unarmed door into InputSigSentState.
The parent commit arms two of the three: the forfeit-bearing transition
through ForfeitSignaturesCollectingState, and the restart reload in
Start. The third is a boarding-only round, which never enters forfeit
collection at all. PartialSigsSentState takes the len(ForfeitMappings)
== 0 branch on OperatorSigned, signs its boarding inputs, and
checkpoints straight into InputSigSentState with an outbox holding only
the sig and registration requests.

That door is the one wavelength#1051 walks through. The round sits at
the same point of no return, but with no reconcile clock the state has
no liveness timer at all: no commitment can confirm, no failure ever
arrives, and a client that never restarts leaves the deposit stranded
until the CSV expires. Only the restart path recovered it, which is why
the DST catch converged and this stayed invisible in review.

The tests assert the invariant rather than the doors, since a fourth
door added later should fail rather than quietly repeat this: whatever
path reaches the checkpoint must arm, and must arm ahead of anything
that can fail on the way out. They also cover the parent's ordering fix
and the disabled opt-out.

Worth recording why no existing test caught it. The boarding harness
leaves StatusReconcileTimeout at zero, which skips the arming branch
outright, so every boarding test that read this outbox saw exactly what
it expected to see.
@Roasbeef
Roasbeef force-pushed the boarding-checkpoint-liveness branch from d70053e to adcea02 Compare August 8, 2026 01:25
In this commit, we add the query that returns a boarding intent adopted
by a dead round to the confirmed pool. It is the inverse of the adopt
write CommitState performs at the checkpoint.

The status reverts to confirmed rather than failed. A dead round proves
the commitment was never broadcast, so nothing on-chain failed and the
UTXO is exactly as it was before the round started. Confirmed is also
the status both recovery routes key on, so restoring it gives the
deposit back to the boardable pool and to the sweep at once.

The update is guarded on the intent still being adopted so a sweep that
has already claimed the deposit is never dragged back.
In this commit, we close the other half of wavelength#1051. Arming the
reconcile clock un-parks the client FSM, but a dead round still has to
return the money, and nothing did: RoundStore had no counterpart to
FinalizeRound, so a round entering ClientFailedState kept its checkpoint
row. reapFailedRounds only drops the in-memory FSM.

Two things stayed stuck as a result. The row sat in ListActiveRounds and
was re-hydrated on every start, re-arming and re-probing a round already
known to be dead. Worse, the boarding intents it adopted stayed adopted,
and boardingIntentSweepable excludes adopted, so the deposit was neither
boardable nor sweepable. Not before the CSV and not after it either,
since nothing writes the expired status. The user saw coins pinned
against the board limit with no way to reach them.

The recovery machinery was already built and simply unreachable.
ListBoardingIntentsByStatus and ListBoardingIntentsBySweepableStatuses
both re-admit a confirmed intent exactly when its linked round reads
failed, and round_statuses has carried a failed row since the schema
landed, commented "Round failed, intents may need recovery". Nothing
ever wrote it.

So FailRound writes it, and reverts the round's adopted intents in the
same transaction. One transaction because the row and the intent
statuses are a single fact about where the deposit lives, and a crash
between the halves would strand it in a round that no longer exists. We
reuse the existing UpdateRoundStatus rather than add a query, since
unlike FinalizeRound this is purely a status move.

The actor calls it from the RoundFailedNotification handler, the one
choke point every failure path already passes through, so no individual
FSM exit has to remember. A round that never reached the checkpoint has
no row and no adopted intents, making both halves no-op updates, so the
call needs no guard. A store error is logged rather than propagated: the
round has already failed and the client has already been told, so a
failed write means the row is reclaimed on a later pass, not that the
failure is in doubt.
@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

@codex Following up on your P1 about dead rounds not being persisted: rather than leave it as a follow-up, we folded the fix into this PR so the whole story reviews as one change. Three new commits.

Your diagnosis was right on both halves, and I verified each independently. RoundStore had no counterpart to FinalizeRound, and boardingIntentSweepable excludes adopted. Worth noting it was worse than "cannot enter CSV recovery": nothing in production writes BoardingStatusExpired either, so an adopted intent on a dead round was never swept at all, CSV or no CSV, and stayed pinned against the board limit via board_limits.go.

The interesting part is that the recovery machinery was already built and simply unreachable. ListBoardingIntentsByStatus and ListBoardingIntentsBySweepableStatuses both re-admit a confirmed intent exactly when its linked round reads failed, and round_statuses has carried a failed row since the schema landed, commented "Round failed, intents may need recovery". Nothing ever wrote that status. So the fix is small: FailRound writes it and reverts the round s adopted intents in the same transaction.

Two judgment calls worth flagging for review. Intents revert to confirmed, not failed, because a dead round proves the commitment never broadcast, so nothing on-chain failed and the UTXO is exactly as it was before the round; confirmed restores that truth and happens to open both recovery routes at once. And the revert is guarded on adopted in SQL so a sweep already in flight is never clobbered, which TestRoundStoreFailRoundLeavesSweptDepositAlone pins.

TestRoundStoreFailRoundReleasesDeposit asserts the deposit is neither boardable nor sweepable before the retirement and both after, so it fails if the retirement is a no-op.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9caa903d47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread round/actor.go
Comment on lines +2798 to +2800
m.RoundID.WhenSome(func(id RoundID) {
a.retireFailedRound(ctx, id)
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Emit retirement on the dead-status transition

Fresh evidence in the updated code shows that the RoundStatusReported(DEAD) branch constructs ClientFailedState and emits only the forfeit rollback/job-drop and timeout cancellation; it never emits a RoundFailedNotification. Consequently this new handler is never reached for the dead-status path it is meant to fix, so FailRound is not called and the checkpoint remains input_sig_sent with its boarding deposits still adopted. Emit the retirement directly from that transition or include a RoundFailedNotification in its outbox.

AGENTS.md reference: round/AGENTS.md:L182-L194

Useful? React with 👍 / 👎.

Comment thread round/transitions.go
RoundKey: RoundKeyStr(s.RoundID.KeyString()),
Phase: TimeoutPhaseStatusReconcile,
Duration: env.StatusReconcileTimeout,
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move the reconcile arm before the timeout cancellation

Fresh evidence in the updated outbox shows that this arm is still appended after the leading CancelTimeoutReq at lines 2889-2894. If the timeout actor rejects that cancellation, processOutbox returns before scheduling this timer even though the FSM checkpoint has already committed InputSigSentState, leaving the round parked without reconciliation until a daemon restart. Put the reconcile arm before the cancellation as well, or make the old-timeout cancellation best-effort.

AGENTS.md reference: round/AGENTS.md:L195-L200

Useful? React with 👍 / 👎.

@litbot-9000

Copy link
Copy Markdown
Collaborator

Pushed two commits addressing the open review threads on 9caa903d. Both are verified locally rather than asserted — details below.

round: retire the round on the checkpointed failure exits (12ff6c9) — addresses @codex's "Emit retirement on the dead-status transition". The finding is correct. retireFailedRound has exactly one caller, the *RoundFailedNotification case in processOutbox, and the dead-answer branch never emits that message: it builds its ClientFailedState by hand, and releaseForfeitsOnFailure returns early on an empty rollback for a boarding-only round. Driving RoundStatusReported(DEAD) through the real FSM before the fix produced an outbox of exactly [CancelTimeoutReq] for a boarding-only round and [ReleaseForfeitReservation, CancelTimeoutReq] with forfeits — no notification either way, so FailRound never ran and the deposit stayed adopted.

TestFailedRoundIsRetiredDurably missed this because it feeds a RoundFailedNotification straight into processOutbox instead of driving the FSM, so it pinned the actor half against a premise — "the single choke point every failure path passes through" — that does not hold for this state. The new tests drive the FSM; all three subcases fail without the fix.

The delivered-failure exit for a round with nothing reserved has the same shape and is fixed alongside it. The BoardingConfirmed error path is deliberately left alone: its commitment confirmed, so retiring there would re-admit intents whose deposit already became a VTXO.

round: arm the reconcile clock ahead of the forfeit cancel (448b3ae) — addresses @codex's "Move the reconcile arm before the timeout cancellation". Also correct, and narrower than it reads: the boarding-only door at PartialSigsSentState already arms first, so wavelength#1051's own path was never exposed; only the forfeit-collection door was. A CancelTimeoutReq is a Tell to the timeout actor like any other — releaseForfeitsOnFailure's comment names it as an entry that can fail mid-flight and short-circuit the outbox, and the disarm site in this same PR already reasons about a saturated timeout actor rejecting one. The arm now leads the whole outbox and the cancel trails it; a leaked one-shot forfeit-collection timer self-loops in InputSigSentState.

assertArmsReconcileFirst carried the exemption that hid this (a cancel "may precede the arm"), so the helper now checks the invariant it claims. Both forfeit-door subcases fail without the reorder.

Still open: @codex's older "Persist dead boarding rounds before leaving reconciliation" thread. That was the same defect from the other end, and 9caa903d plus these two commits close it — the persisted checkpoint is retired and the adopted intents reverted on the dead path now. Worth resolving that thread rather than leaving it hanging.

On the red CI: infrastructure, not the code. All four genuine failures on run 31234528204 (linux-armv7 cross-compile, system tests (postgres), unit tags="test_postgres", unit-cover) carry the check-run annotation "The self-hosted runner lost communication with the server" on runner deployment 44c8q, with every step from the build/test step onward at a null conclusion — ARC runners getting preempted. The three darwin/windows cross-compile failures are matrix cascade: "The strategy configuration was canceled because cross-compile.linux-armv7_linux-armv7 failed". unit-race passed at 17m51s on that same run.

Verified locally: go test ./round/ and go test -race ./round/ pass on both commits (checked the intermediate commit separately, so the series bisects clean), go vet ./round/ clean, gofmt clean, make commitmsg-lint OK on both.

@codex please re-review the two commits above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review: round: arm the status-reconcile clock for every checkpointed round

Reviewed the full diff, the FSM transition changes, the DB layer, and the test suite. This is a carefully-reasoned fix and the commit messages are exemplary — the invariant ("the reconcile clock is armed for the whole of InputSigSentState, and every exit disarms it") is clearly stated, encoded in the docs, and pinned by position-based tests. Overall this looks correct and ready. A few observations, mostly for robustness and awareness rather than blockers.

Correctness — looks solid

  • The 'adopted' guard on RevertAdoptedBoardingIntent is the right call and is exactly what protects against the dangerous case: a deposit that already became a VTXO (status no longer adopted) or an in-flight sweep (sweep_pending) is never dragged back. TestRoundStoreFailRoundLeavesSweptDepositAlone covers this well.
  • Reverting to confirmed rather than failed is well-justified — a dead round proves the commitment never broadcast, so the UTXO is unchanged. Good that this also restores both recovery routes at once.
  • context.WithoutCancel in retireFailedRound is correct — the write must outlive the request that triggered it.
  • I verified the forfeit-collection cancel now trailing the reconcile arm is safe: a ForfeitCollectionTimedOut arriving after the transition reaches InputSigSentState hits the default: return selfLoop(s) case (transitions.go:5026), so it self-loops harmlessly as the commit claims.
  • Confirmed InsertRound only runs in CommitState (the checkpoint), so FailRound on a pre-checkpoint round is genuinely a no-op — no risk of clobbering a round that hasn't checkpointed.

Suggestion — consider guarding the round-status write (defense-in-depth)

FailRound's intent revert is guarded on status = 'adopted', but the round-status write is unconditional:

-- UpdateRoundStatus
UPDATE rounds SET status = $2, last_update_time = $3 WHERE round_id = $1;

Retirement now fires on every RoundFailedNotification, which flows through many failure paths (failWithNotification, failureOutbox, and the two checkpointed exits). Today no post-confirmation state emits RoundFailedNotification — I checked ConfirmedState.ProcessEvent, which only handles RoundComplete and otherwise self-loops, and the BoardingConfirmed error path deliberately emits only a disarm — so there is no path that would move a confirmed round back to failed. This is safe as written.

However, the round-row half relies on that whole-FSM invariant staying true, while the intent half is locally defended by its WHERE status = 'adopted'. If a future exit ever emits a failure notification for an already-confirmed round, the intents would be protected but the round row would silently flip to failed. A WHERE round_id = $1 AND status = 'input_sig_sent' guard on the retirement write (or a dedicated query, rather than the generic UpdateRoundStatus) would make the two halves symmetric and the safety local rather than global. Optional, but cheap insurance given how many paths now reach this call.

Minor notes (non-blocking)

  • Write per failure: FailRound now opens a write transaction (GetRoundBoardingIntents + UpdateRoundStatus) on every failure notification, including frequent pre-checkpoint recoverable failures where both are no-ops. The cost is negligible and the unconditional design keeps the actor simpler, so this is a reasonable trade — just flagging that a failure storm now costs one write-txn each.
  • Metric semantics: emitRoundCompleted(..., \"failed\") now also fires on the dead-answer and delivered-failure boarding paths that previously emitted no RoundFailedNotification. This looks intended (the counter now covers these strands), but worth confirming the failed-round counter's meaning shift is expected downstream.

Tests

Coverage is thorough and, notably, the ordering assertions check position not just presence (assertArmsReconcileFirst, outboxIndexOf, the disarm-trails-delivery checks), which is exactly what this fix needs since processOutbox abandons the outbox on the first failing Tell. The three-door / every-exit framing in checkpoint_arming_test.go should catch a future fourth door that forgets to arm. Nice touch documenting why the boarding harness's zero StatusReconcileTimeout hid the original gap.

Nice work — the fix is tight and the reasoning is a pleasure to follow.

In this commit, we make the retirement added in the previous commit
actually reachable. Retirement hangs on RoundFailedNotification in the
actor, and the comment on TestFailedRoundIsRetiredDurably states the
premise it relies on: that the notification is the single choke point
every failure path passes through. That is true of the paths built by
failWithNotification and failureOutbox, and false of the two exits out
of InputSigSentState, which assemble their transitions by hand.

The dead-answer branch is the one that matters. It builds a
ClientFailedState directly and hands it to releaseForfeitsOnFailure,
which prepends the rollback and, for a boarding-only round, returns
early on an empty one. So the outbox that leaves the authoritative dead
verdict is a lone CancelTimeoutReq: the FSM fails in memory while the
checkpoint row stays in ListActiveRounds and the deposits it adopted
stay adopted, out of the sweep and pinned against the board limit. That
is the wavelength#1051 strand the reconcile exists to end, still open on
the exact path the reconcile added.

The delivered-failure exit for a round with nothing reserved has the
same shape and the same consequence, so it emits the notification too.
Both keep the disarm trailing, since a cancel ahead of a delivery lets a
saturated timeout actor suppress that delivery.

The BoardingConfirmed error path is deliberately left alone: its
commitment confirmed, so retiring the round would re-admit intents whose
deposit already became a VTXO.

The tests drive the FSM rather than feeding the notification to
processOutbox by hand, which is what let the gap through: all three
subcases fail without the fix.
In this commit, we close the last gap in the arming invariant. The
forfeit-collection door arms the clock ahead of the sig and registration
requests, on the reasoning that processOutbox abandons the outbox at the
first failing Tell and the FSM has already checkpointed by dispatch
time. The cancel of the forfeit-collection timeout still led the outbox,
though, and a cancel is a Tell to the timeout actor like any other: the
comment on releaseForfeitsOnFailure names CancelTimeoutReq as an entry
that can fail mid-flight and short-circuit the rest, and this PR's own
disarm site orders around a saturated timeout actor rejecting one. A
rejection there commits the checkpoint with no clock until a restart,
which is the strand the arm exists to prevent.

So the arm leads the whole outbox and the cancel trails it. Trailing
costs nothing: a cancel that never lands leaks a one-shot timer, and a
forfeit-collection timeout arriving after the transition reaches
InputSigSentState, which has no case for it, self-loops.

assertArmsReconcileFirst carried the exemption that hid this, treating a
cancel as bookkeeping that may precede the arm. It no longer does, so
the invariant the helper claims is the one it checks: both forfeit-door
subcases fail without the reorder.
@litbot-9000
litbot-9000 force-pushed the boarding-checkpoint-liveness branch from 448b3ae to 3eaa7fd Compare August 11, 2026 00:48
@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review: round: arm the status-reconcile clock for every checkpointed round

Reviewed all 7 commits. This is a careful, well-motivated fix for the two halves of #1051 — the parked FSM (arm the reconcile clock on every door into InputSigSentState) and the pinned deposit (FailRound retires the dead round and un-adopts its intents). The commit messages and the new Invariants entries in round/CLAUDE.md/AGENTS.md are excellent, and the ordering discipline (arm leads every fallible Tell, disarm trails every delivery) is applied consistently and asserted by position rather than presence. Tests are thorough and each case is claimed to fail against pre-fix code.

A few observations, none blocking.

1. FailRound's round-status write is unguarded, unlike the intent revert (defensive)

RevertAdoptedBoardingIntent is correctly guarded on status = 'adopted', so a deposit a sweep already claimed is never dragged back. But the companion UpdateRoundStatus is WHERE round_id = $1 with no status predicate, so it unconditionally stamps failed.

Today the FSM's terminal-state guarantees mean RoundFailedNotification can't fire for an already-confirmed round, so this is safe in practice. But it's an asymmetry: the intent half is race-hardened and the round half isn't. Consider scoping the write to the checkpoint status — e.g. ... AND status = 'input_sig_sent'. That makes FailRound precisely idempotent, keeps it a strict no-op for a pre-checkpoint round, and makes a late/duplicate failure structurally unable to clobber a finalized row. ListActiveRounds already keys on status = 'input_sig_sent', so the scoped predicate matches exactly the rows this is meant to retire.

2. retireFailedRound runs a write transaction on every failure path (perf, acknowledged tradeoff)

The commit deliberately makes retirement unconditional across failure paths, relying on both halves being no-op UPDATEs for a round that never checkpointed. That's a reasonable simplicity call, but note it adds a GetRoundBoardingIntents + UpdateRoundStatus write transaction, synchronously in the actor loop, to every admitted-round failure — including the common early failures (admission timeout, quote rejection) that never reached the checkpoint and have no row to retire. Round failures are rare enough that this is likely fine, but it is real write-transaction churn on the hot failure path where a cheap in-memory "was this checkpointed?" check would elide it. Worth a conscious "keep it simple" decision rather than an accident.

3. Test coverage — one worthwhile addition

The suite covers the three arm doors, the disarm exits, both orderings, the SQL guard, and the actor retirement + store-error tolerance — strong. The one invariant not locked by a test is the negative case the commit reasons about explicitly: the BoardingConfirmed → VTXO-build-error exit must not emit RoundFailedNotification (retiring a round whose commitment already confirmed would re-admit intents whose deposit already became a VTXO). Since retirement now hangs entirely on that notification, a regression test asserting this path emits a disarm but no RoundFailedNotification would pin the "deliberately left alone" reasoning against a future refactor that routes it through checkpointedFailureOutbox.

Nits

  • checkpointedFailureOutbox drops evt.FailureCode (the notification struct has no such field), while the resulting ClientFailedState still carries it. That matches the existing failWithNotification shape, so it's consistent — just confirming it's intentional that the typed code stays on the state and not the notification.
  • The sqlc param↔placeholder mapping for RevertAdoptedBoardingIntent checks out ($1/$2 = outpoint, $3 = last_update_time).

Overall this is a solid fix with the reasoning captured durably in-repo. The status-write guard (#1) is the one item I'd genuinely consider before merge; the rest are optional.

litbot-9000 and others added 2 commits August 11, 2026 01:22
In this commit, we make FailRound structurally unable to retire a round
that is not checkpointed, closing the gap between the two halves of the
retirement.

The intent half was already guarded on 'adopted', so a deposit some
other path had claimed was never dragged back. The round half went
through UpdateRoundStatus, which is keyed on round_id alone and would
stamp 'failed' over any status at all.

That asymmetry matters more than it looks, because the round row is not
bookkeeping: it is the gate. Both re-admission queries decide whether an
adopted deposit is boardable or sweepable by joining through
round_boarding_intents to the round row and asking whether it reads
'failed'. A confirmed round is exactly what keeps a deposit out of those
pools once its commitment is on-chain and the UTXO has become a VTXO.
Retiring such a round would therefore not merely mislabel a row, it
would offer an already-spent outpoint back for a fresh board.

Nothing produces that ordering today, since the FSM cannot fire
RoundFailedNotification for a round it has already seen confirm. This is
a structural guard, not a fix for a live bug, and the point is that a
future caller cannot reintroduce the hazard by accident.

We add a new query rather than tightening UpdateRoundStatus, which is a
generic setter exposed on both sqlc.Querier and db.RoundStore. Silently
narrowing it would turn every other status move into a conditional
no-op under a name that promises otherwise, which is a worse trap than
the one being closed.

RetireCheckpointedRound is guarded on 'input_sig_sent', the exact status
ListActiveRounds keys on, and returns its row count. FailRound leads
with it and gives the deposits back only if it actually consumed a row,
so a failure that races a confirmation touches neither half.

The sqlc stubs were regenerated by hand because Docker is unavailable in
this environment; CI's sqlc-check is the authority on them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In this commit, we add a test asserting an absence: the
BoardingConfirmed -> buildClientVTXOs error exit in InputSigSentState
must emit the reconcile disarm and must NOT emit a
RoundFailedNotification.

Retirement now hangs entirely on that notification, and the two exits
either side of this one both emit it. That makes this branch look like
an oversight, and routing it through checkpointedFailureOutbox looks
like a tidy unification of the ClientFailedState exits.

It is not. The commitment already confirmed on this path. The adopted
deposits have become VTXOs, and the round row is precisely what keeps
them out of the boardable and sweepable pools, since both re-admission
queries join through it and re-admit only on 'failed'. Retiring the
round would hand back deposits the client no longer holds as UTXOs and
offer already-spent outpoints for a fresh board. What failed here is
local bookkeeping; the round itself succeeded.

The disarm is still correct and still asserted: the confirmation
resolved the round's fate, so a one-shot left armed would fire a probe
at a round that has settled terminally.

Nothing pinned either half of that reasoning, so a refactor could have
made the change silently. Both assertions were verified to fail against
a mutated transition: routing the branch through
checkpointedFailureOutbox trips the retirement check, and dropping
reconcileDisarmEvents trips the disarm check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review: arming the status-reconcile clock for every checkpointed round

I read through all 9 commits, the FSM exits in round/transitions.go, the actor
retirement path in round/actor.go, and the new db query + store logic. This
is a genuinely careful piece of work — the diagnosis (boarding-only rounds have
no liveness clock in InputSigSentState because every reconcile site gated on
a non-empty forfeit set) is precise, and the fix reduces to a cleaner invariant
than the one it replaces. Notes below, mostly confirmations with a few things
worth a second look.

What's strong

  • Invariant framing is sound and now documented where it's enforced. "Every
    door arms, every exit disarms; the arm leads the fallible sends, the disarm
    trails every delivery" is the right way to think about a checkpointed liveness
    clock, and mirroring it into round/CLAUDE.md + AGENTS.md means the next
    person to add a fourth door has a spec to violate.
  • Ordering is asserted by position, not presence (outboxIndexOf,
    assertArmsReconcileFirst). Since processOutbox abandons the outbox on the
    first failing Tell, presence-only tests would pass while the actual hazard
    (arm behind a fallible send, or cancel ahead of a delivery) went uncaught.
    Patch 7 tightening assertArmsReconcileFirst to also reject a cancel ahead
    of the arm — a CancelTimeoutReq is a Tell to the same saturated actor — is
    the subtle case most reviewers would have waved through.
  • The two-halves split (un-park the FSM / give the deposit back) is the right
    decomposition,
    and the RetireCheckpointedRound guard on
    status = 'input_sig_sent' (patch 8) is a real structural improvement over the
    generic UpdateRoundStatus: it makes "retire a round that already confirmed"
    impossible rather than merely unreached, so a future caller can't re-admit
    deposits that are already VTXOs. TestConfirmedThenFailedDoesNotRetireTheRound
    pinning that absence is exactly the test that keeps a later "tidy
    unification" refactor from silently regressing it.

Things worth confirming

  1. New RoundFailedNotification on the forfeit-bearing dead-answer path
    changes metrics/notification emission.
    After patch 6, the dead-answer exit
    for a forfeit-bearing round emits both RoundFailedNotification and
    TerminalJobFailedNotification. The former newly triggers
    emitRoundCompleted(…, "failed") on a path that previously emitted no
    round-completed metric at all. This looks correct (the round did fail), and
    handleTerminalJobFailure doesn't also count, so there's no double-count —
    but it is a behavior change for any downstream consumer that keys on these
    two notifications. Worth a sentence confirming nothing treats the pair as two
    distinct round failures.

  2. Delivered-failure branch with forfeits present and reconcile disabled.
    The len(s.Intents.Forfeits) == 0 || env.StatusReconcileTimeout <= 0 branch
    (transitions.go ~4658) now emits RoundFailedNotificationFailRound
    reverts adopted boarding intents, but it still does not release the
    forfeit VTXO reservations. For a mixed boarding+forfeit round with reconcile
    turned off (negative timeout), the boarding deposit comes back while the
    forfeit reservations stay stranded. This is a pre-existing gap (the branch
    never released here), and reconcile-disabled is a config edge, so it's narrow
    — but the PR now makes the asymmetry more visible (retire durable round +
    revert boarding, yet leak the reservation). A one-line comment or a follow-up
    issue would help the next reader not mistake it for an oversight introduced
    here.

  3. FailRound now opens a write transaction on every RoundFailedNotification.
    Because retirement hangs off the single choke point, pre-checkpoint recoverable
    failures also call it, where RetireCheckpointedRound matches 0 rows and
    returns early — a guaranteed no-op, but still a DB round-trip per failure.
    Failures are rare so this is fine; just flagging it's no longer free.

  4. sqlc stubs were hand-regenerated (Docker unavailable, per the patch-8
    message). The generated querier.go / round.sql.go match the .sql by
    inspection, but CI's sqlc-check is the authority — worth a green check
    before merge.

Nothing here blocks; (1) and (2) are confirmations more than defects. Nicely
scoped fix for #1051.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v0.1.x-branch Backport this merged PR to v0.1.x-branch claude-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

round: boarding-only round recovered from checkpoint has no liveness clock, strands the deposit

2 participants