You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a8afa20
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/adr/0015-runtime-coordination.md
+16-9Lines changed: 16 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,10 @@ Four pieces, decided together. Two are runtime-local (admission bound, burst sha
31
31
32
32
Add `MaxDetached int` to **Runtime Options**: a per-instance cap on admitted-but-incomplete deferred dispatches. Everything a deferred delivery retains counts against it — running tails, queue/debounce parked waiters, concurrent slot-waiters, and (once burst ships) batch members — from admission until the dispatch reaches a terminal outcome. Deliveries that resolve in the prelude (duplicate, dropped conflict, ignored, unrouted, error) release their slot at acknowledgement; only routed deferred work holds it to the tail's end.
33
33
34
-
Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to the platform's retry-inducing response (HTTP 429/503 equivalents; the mapping is adapter-owned). Because the delivery was never acknowledged and never marked, the platform's own retry redelivers it and is not deduped away — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away).
34
+
Semantics at the cap: **reject-with-signal, before ack and before dedupe marking.** Dispatch fails fast with a typed sentinel (`ErrAdmissionRejected`) before the delivery is marked in **Event Identity** dedupe; the webhook layer maps it to a shape-aware response (adapter-owned). Because the delivery was never acknowledged and never marked, no dedupe record blocks a retry — the same honesty contract prelude errors already follow (a failed prelude leaves the event un-marked so a retry is not deduped away). The mapping is shape-aware because platforms do not retry every webhook shape:
35
+
36
+
-**Platform-retried deliveries** (e.g. Slack Events API callbacks): the adapter returns the retry-inducing response (HTTP 429/503 equivalents) and the platform's own redelivery covers the event.
37
+
-**Direct user invocations that the platform does not redeliver** (e.g. Slack slash commands and interactivity — the in-repo Slack normalization records that these carry no retry headers): a bare 429 would turn a click into a silent permanent failure, so the adapter must answer with a *truthful busy response* to the user (a visible "busy, try again" acknowledgement, observed as rejected) — never a silent failure, never a fake success. The user's own retry is honored because no dedupe record exists.
35
38
36
39
Placement and validation:
37
40
@@ -83,7 +86,8 @@ Fences come from **one global, non-resetting sequence**, not per-scope counters:
83
86
84
87
Protocol — for `queue` and `debounce` under `DispatchDeferred` only. Burst is delivery-preserving and drop is first-wins, so neither coalesces by fence. Synchronous queue waits are excluded by design: a sync park is bounded only by the caller's request context, which need not carry a deadline, so no finite park horizon exists from which to derive a sound registration TTL (see Non-goals). Under deferred dispatch every park is bounded by `DetachTimeout`.
85
88
86
-
- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within one `ThreadLockTTL` of registration (both bounds are runtime-enforced, not assumed). Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran.
89
+
- **Allocate early; register within a bounded window.** A routed queue/debounce delivery allocates its fence before the reorderable prelude work (the dedupe mark and lock acquisition — the State round-trips where deliveries actually stall). This is the cross-instance analog of the local admission sequence: a delivery delayed in its prelude can never displace a waiter fenced after it (class 7). A fence is registrable only within a bounded registration window from its allocation — one `ThreadLockTTL`, enforced by the runtime on a local monotonic clock — and the runtime likewise requires the dispatch-time check to begin within `ThreadLockTTL` + `DetachTimeout` of registration: one lock TTL of tail-start slack plus the full `DetachTimeout`-bounded park, so an ordinary long queue wait (a holder running through many lease refreshes, within `DetachTimeout`) stays fenced for its entire wait. Both bounds are runtime-enforced, not assumed. Past either bound, a marked delivery proceeds per-instance (unfenced or degraded, observed) and an unmarked one fails the prelude, so the platform retry redelivers with a fresh fence. Without this bound a delivery stalled between allocation and registration could outlive the register entries of newer, already-completed deliveries and register its stale fence into an empty register — dispatching an older event after a newer one ran.
90
+
-**One ordering source.** When the capability is present, the **Waiter Fence***is* the admission order: local supersession among fenced deliveries compares fences, not the local admission sequence, so local and global selection can never diverge. (Two deliveries interleaving fence allocation on one instance could otherwise be ordered oppositely by sequence and by fence — the local registry displacing one while the register refuses the other, skipping both turns.) The local admission sequence orders only unfenced deliveries — capability absent, sync dispatch, or degraded — and a mix of fenced and degraded deliveries follows today's per-instance rules with per-instance honesty.
87
91
-**Every routed delivery registers.** Whether it parks or acquires the **Thread Lock** immediately, a routed queue/debounce delivery registers its fence; only duplicates, unrouted events, and prelude failures never register (so they can never displace a live waiter). Registering the immediate acquirer is what closes the barging hole: a newer delivery that wins the lock race against a parked older waiter publishes its fence, so the older waiter observes it at dispatch time and skips — an older event can never dispatch *after* a newer one ran. `RegisterWaiter` returning false means a newer waiter is already registered: the delivery is superseded and skips immediately (releasing any lease it holds), exactly like local supersession.
88
92
-**Check on dispatch — the turn-claim point.** Every registered delivery (including a preemptor — §4) re-reads `NewestWaiter` once, after acquiring the **Thread Lock** and immediately before running the handler. If a newer fence is registered, it skips: release the lock, emit the observable skip. Once a handler starts it is never retroactively superseded — a follow-up arriving after the turn is claimed queues behind it, matching local queue semantics.
89
93
@@ -95,7 +99,7 @@ The guarantee split, stated honestly:
95
99
96
100
Failure modes accepted and documented, not hidden:
97
101
98
-
- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the tail-start bound plus `DetachTimeout` — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both leading intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
102
+
- **Register-then-never-run loses the coalesced turn.** If the instance holding the newest fence crashes, shuts down, or abandons its delivery (`DetachTimeout`) after older waiters skipped, no instance dispatches that group's turn — where per-instance coalescing would have dispatched a stale event. This is the coordination-only price: takeover would require the event payload in State (a message store — refused). Register entries carry a runtime-derived TTL: the registration window plus the check horizon — that is, 2×`ThreadLockTTL` + `DetachTimeout` — with a safety margin; `DetachTimeout` alone is insufficient because the detach clock starts only when the tail does, and both intervals are the runtime-enforced bounds above, so implementations and operators can compute the TTL from configuration. This arithmetic makes the ordering sound: while any older fence remains registrable, every newer registration that could supersede it is still visible, and a parked waiter always finds a newer turn-claimer's registration alive at its dispatch-time check. Expiry beyond that horizon degrades to per-instance semantics rather than blocking dispatch. The loss window is the same class ADR 0002 already accepts for deferred dispatch (a crash after ack loses the work).
99
103
- **Coalescer failures degrade; they never lose an accepted event.** The dedupe mark is the commit point of the prelude. Any `Coalescer` failure during the prelude — allocation or registration, whether a backend error or the request context ending — degrades that delivery to per-instance semantics (it proceeds unfenced or unregistered, observed); it never fails the prelude and never abandons the delivery. A marked-but-unacknowledged delivery is therefore never silently dropped by a coordination call, preserving ADR 0002's retry contract (a platform retry of an un-marked failure redelivers; a marked delivery always launches). The post-ack dispatch-time check splits differently: a backend error degrades to per-instance semantics for that dispatch, while cancellation or deadline expiry of the delivery's own context is the existing abandonment path (exit without running) — a delivery whose execution budget ended can never "fall back" into running the handler past its budget. Events are never lost to a register outage — they serialize under the **Thread Lock** as today.
100
104
101
105
Backend fit — all four in-repo States implement it, and the conformance suite grows a capability section: **Memory State** (atomic counter + per-scope register map with lazy expiry), **Redis State** (one persistent `INCR` key + per-scope compare-greater script with TTL), **Postgres State** (a sequence + per-scope register row with expiry), **NATS State** (the revision of a single allocation key for fences; a register bucket whose bucket-level TTL follows ADR 0014's uniform-TTL mechanics — the adapter validates and does not honor the per-call `ttl`, so its operator-configured register-bucket TTL must cover the fleet maximum of the derived register TTL (2×`ThreadLockTTL` + `DetachTimeout` + margin); mixed-configuration fleets size it to the maximum). Third-party States keep compiling and keep today's semantics.
@@ -115,16 +119,19 @@ type LockHolder struct {
115
119
116
120
// LockForcer is an optional State capability enabling preemption.
117
121
typeLockForcerinterface {
118
-
// ObserveLock reports the current holder of key, if held.
122
+
// ObserveLock reports the current holder of key, if held. A holder
123
+
// installed via TakeLock must be recognizable to its caller through
124
+
// ObserveLock (identity equal to, or a specified derivation of, the
-**Takeover, not release-then-acquire.**`TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. `ObserveLock`/`TakeLock`*failures* follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched.
134
+
- **Takeover, not release-then-acquire.** `TakeLock` is a single compare-and-swap on the lock key: there is no window in which a third party can acquire between "release" and "acquire" — the class-3 successor race is precluded by construction, not by runtime choreography. A holder's ordinary `ExtendLock` churn (e.g. NATS revision bumps) must not defeat a takeover whose observed `Identity` still matches; a genuine holder change must. On `taken=false` the preemptor falls back to the configured strategy path, observed — no retry loop. The replacement **Lock Lease** token is generated by the caller and passed in, making takeover idempotent: an *ambiguous* outcome — the backend committed the swap but the response was lost — is reconciled by re-observing the lock, and a holder matching the caller's replacement token means the takeover committed (the preemptor proceeds holding it) rather than an orphaned lock stalling the scope until TTL. Only a confirmed non-commit is classified as a failure, and `ObserveLock`/`TakeLock` *failures* then follow §3's commit-point rule: after the dedupe mark, a backend error or the request context ending falls back observably to the configured strategy path — it never abandons the delivery and never fails the prelude post-mark, so a platform retry can never be deduped away from an event that was never launched.
128
135
-**No local preemption registry.** The staged `inflightCancels` map, `preemptLocalIfPending`, and the `victimDone` handshake are rejected wholesale (classes 4 and 5). Preemption takes exactly one path — through the State — whether the victim is local or remote. The victim is cancelled via the universal lease-loss cancellation already on main: its next `ExtendLock` fails and its **Detached Work Context** is cancelled with `ErrPreempted`. Preemption additionally requires each lease-refresh call to be independently time-bounded, with a refresh that cannot complete within its bound counting as lease loss: today's loop calls `ExtendLock` under `context.WithoutCancel` with no RPC deadline, so a backend that stalls after a takeover would otherwise delay the victim's cancellation signal indefinitely. With that bound, the signal latency is at most one refresh interval plus the per-call bound.
129
136
-**What preemption means, honestly:** preemption invalidates the victim's lease; it does not stop the victim. The victim's context is cancelled within one refresh interval plus the per-call bound above (`ThreadLockTTL/2`; ~60s at defaults), but termination is cooperative — Go cannot forcibly stop a handler — so victim and preemptor overlap for the cancellation-signal latency plus however long the victim takes to honor its context, and a handler that ignores cancellation overlaps unboundedly. This is the same cooperative-cancellation contract every runtime stop already carries (`DetachTimeout`, shutdown drain, lease loss); preemption adds no stronger guarantee. `ThreadLockTTL` tunes the signal latency: shorter TTL, faster cancellation, more State load. A future local-victim nudge is permitted as a best-effort latency optimization only — it must never carry correctness (no registry the semantics depend on).
130
137
-**Hook and gating.**`RuntimeOptions.OnLockConflict LockConflictHook` (`func(context.Context, *Event) bool`) is consulted on a **Lock Conflict**; true → observe + take, false → the configured strategy path. Constructor validation requires `DispatchDeferred` (only deferred holders participate in lease-loss cancellation; a sync holder cannot be stopped — the class-2 mixed-fleet finding), requires the State to implement `LockForcer` (a preemption hook on a State that cannot preempt is a misconfiguration, failed fast, not degraded silently), and rejects `ConcurrencyConcurrent` (that strategy takes no **Thread Lock**, so no **Lock Conflict** can ever invoke the hook; accepting the combination would silently disable a configured preemption policy).
@@ -173,7 +180,7 @@ This design explicitly refuses to promise:
173
180
## Consequences
174
181
175
182
- Two new **Runtime Options** (`MaxDetached`, `MaxBurstBatch`), one new sentinel (`ErrAdmissionRejected`), and one observation/outcome pair for admission. `MaxDetached` positive is required under `DispatchDeferred`: existing deferred configurations built without `DefaultRuntimeOptions` must set it. Deliberate — no unbounded production default survives this ADR.
176
-
- Adapters gain one honesty duty: map `ErrAdmissionRejected`to the platform's retry-inducing response. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc.
183
+
- Adapters gain one honesty duty: map `ErrAdmissionRejected`shape-awarely — a retry-inducing response for platform-redelivered shapes, a truthful busy response for direct invocations the platform does not redeliver. Sustained rejection can trip platform webhook-health policies; the alternative (acknowledging discarded work) is worse, and sizing guidance lives with the option's GoDoc.
177
184
- The `State` interface itself does not change; both extensions are optional capabilities (ADR 0009 precedent). All four in-repo States implement both; the conformance suite grows capability sections (fence monotonicity, register-only-newer, TTL expiry, observe/take atomicity under contention, extend-churn non-defeat).
178
185
- Queue/debounce dispatch under `DispatchDeferred` gains up to three State round-trips (allocate, register, check) when `Coalescer` is present — the cost of global newest-wins. Absent the capability, or under sync dispatch, nothing new is paid.
179
186
- ADR 0012's proposed force surface is superseded by §4 (flagged per the domain docs' ADR-conflict rule). ADR 0012's status note and the CONTEXT.md glossary (**Admission Bound**, **Waiter Fence**, **Lock Takeover**) update when this ADR is accepted and implementation lands.
0 commit comments