Bug: run with no eligible worker waits silently forever (no fail-fast on permanent dead-end)
Summary
When selectWorker returns null, the engine treats every case identically: retry with backoff. It cannot distinguish:
- transient backpressure — matching workers exist but are saturated → retrying is correct;
- permanent dead-end — no active worker matches the tags at all (wrong tag, worker dead/drained, never registered) → retrying loops forever.
In the permanent case the run sits in running state indefinitely with no terminal failure, no surfaced error, and repeated retry log spam.
Root cause
src/dispatch.zig:37 selectWorker returns null for both conditions; the engine's retry loop has no way to ask "does any active worker ever match these tags?".
Reproduction
- Submit a run whose step requires
worker_tags: ["nonexistent-tag"].
- Run never terminates; steps retry forever;
/runs/{id} shows running with repeated attempts.
Expected behavior
A hasActiveWorkerForTags(tags)-style check (active status + tag intersection, ignoring load) distinguishes the two cases:
- matching active worker exists but is saturated → keep retrying (backpressure);
- no matching active worker → fail the run with a clear error ("no worker can serve node X (tags [...])").
This also matters for saturation policy: a single saturated worker with max_concurrent=1 should not be mistaken for a dead-end.
Bug: run with no eligible worker waits silently forever (no fail-fast on permanent dead-end)
Summary
When
selectWorkerreturns null, the engine treats every case identically: retry with backoff. It cannot distinguish:In the permanent case the run sits in
runningstate indefinitely with no terminal failure, no surfaced error, and repeated retry log spam.Root cause
src/dispatch.zig:37selectWorkerreturnsnullfor both conditions; the engine's retry loop has no way to ask "does any active worker ever match these tags?".Reproduction
worker_tags: ["nonexistent-tag"]./runs/{id}showsrunningwith repeated attempts.Expected behavior
A
hasActiveWorkerForTags(tags)-style check (active status + tag intersection, ignoring load) distinguishes the two cases:This also matters for saturation policy: a single saturated worker with
max_concurrent=1should not be mistaken for a dead-end.