You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit aba6b2a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/adr/0015-runtime-coordination.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -95,10 +95,10 @@ The guarantee split, stated honestly:
95
95
96
96
Failure modes accepted and documented, not hidden:
97
97
98
-
-**Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL no shorter than the deferred park horizon (`DetachTimeout`); expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
99
-
-**Backend unavailability degrades; the delivery's own cancellation does not.** A `Coalescer` backend error falls back to per-instance semantics for that dispatch, with an observation; events are never lost to a register outage — they serialize under the **Thread Lock** as today. Cancellation or deadline expiry of the delivery's own context is *not* a degradation case: it is the existing abandonment path (exit without running), so a delivery whose execution budget ended at the dispatch-time check can never "fall back" into running the handler past its budget.
98
+
-**Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a TTL sized to the delivery's full possible park: `DetachTimeout`**plus** the bounded interval between registration (which happens in the request-context prelude) and tail start, with a safety margin — `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does. Expiry degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
99
+
- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today.
100
100
101
-
Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key + register keys). Third-party States keep compiling and keep today's semantics.
101
+
Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register TTL must cover the fleet's maximum park horizon, and mixed-`DetachTimeout` fleets size it to the maximum). Third-party States keep compiling and keep today's semantics.
102
102
103
103
### 4. Preemption on a fenced Lock Takeover (`LockForcer` reshaped; issue #50's scope extension)
104
104
@@ -127,7 +127,7 @@ type LockForcer interface {
127
127
-**Takeover, not release-then-acquire.**`TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop.
128
128
-**No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted` within one refresh interval.
129
129
-**What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval (`ThreadLockTTL/2`; 60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on).
130
-
-**Hook and gating.**`RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding) and requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently).
130
+
-**Hook and gating.**`RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy).
131
131
-**Stale preemptors cannot dispatch stale events.** A preemptor is a conflicted delivery: under queue/debounce it holds a fence like any routed delivery and performs the §3 dispatch-time check after `TakeLock`. If a newer waiter registered meanwhile, the preemptor skips and releases. Residual, documented: a preemptor superseded between its conflict decision and its takeover may still invalidate a victim whose successor would have queued — a bounded spurious preemption, observable, but never a stale dispatch and never an innocent successor's lease deleted.
132
132
-**Mixed fleets:** preemption assumes instances sharing a State configure it uniformly. A `DispatchSync` holder whose lease is taken keeps running until its handler returns (it has no refresh loop) — the same overlap risk sync dispatch already carries for TTL expiry today. Documented, not solved here.
0 commit comments