Skip to content

BUG: run with no eligible worker waits silently forever (no fail-fast on permanent dead-end) #44

Description

@addadi

Bug: run with no eligible worker waits silently forever (no fail-fast on permanent dead-end)

Summary

When selectWorker returns null, the engine treats every case identically: retry with backoff. It cannot distinguish:

  • transient backpressure — matching workers exist but are saturated → retrying is correct;
  • permanent dead-end — no active worker matches the tags at all (wrong tag, worker dead/drained, never registered) → retrying loops forever.

In the permanent case the run sits in running state indefinitely with no terminal failure, no surfaced error, and repeated retry log spam.

Root cause

src/dispatch.zig:37 selectWorker returns null for both conditions; the engine's retry loop has no way to ask "does any active worker ever match these tags?".

Reproduction

  1. Submit a run whose step requires worker_tags: ["nonexistent-tag"].
  2. Run never terminates; steps retry forever; /runs/{id} shows running with repeated attempts.

Expected behavior

A hasActiveWorkerForTags(tags)-style check (active status + tag intersection, ignoring load) distinguishes the two cases:

  • matching active worker exists but is saturated → keep retrying (backpressure);
  • no matching active worker → fail the run with a clear error ("no worker can serve node X (tags [...])").

This also matters for saturation policy: a single saturated worker with max_concurrent=1 should not be mistaken for a dead-end.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions