You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 252b84f
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/adr/0015-runtime-coordination.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -99,7 +99,7 @@ The guarantee split, stated honestly:
99
99
100
100
Failure modes accepted and documented, not hidden:
101
101
102
-
- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
102
+
- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. The arithmetic assumes fleet-uniform `ThreadLockTTL` and `DetachTimeout` — the same uniform-fleet operator contract as capability participation: a short-timeout instance's registration could otherwise expire while a long-timeout instance's older fence is still within its horizon, reopening the empty-register reordering. Heterogeneous fleets must therefore derive every backend's registration TTL from fleet-wide maxima — for the per-call-TTL backends (Memory, Redis, Postgres) as much as for NATS's bucket TTL. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
103
103
- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today.
104
104
105
105
Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics.
@@ -131,7 +131,7 @@ type LockForcer interface {
131
131
}
132
132
```
133
133
134
-
- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. Only a confirmed non-commit is classified as a failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched.
134
+
- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. The reconciliation read is itself fallible, so it must be retried — reads are idempotent — on an independently bounded context until the replacement is confirmed present or definitively absent, or a bounded reconciliation budget expires; an outcome still inconclusive after the budget falls back with its own observation and is a documented backend-outage residual (the orphaned replacement expires at TTL, and under `drop` both the victim's and the preemptor's turns can be lost — the same outage class as a register failure). Only a confirmed non-commit is classified as an ordinary failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched.
135
135
-**No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound.
136
136
-**What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on).
137
137
-**Hook and gating.**`RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy).
0 commit comments