Bug: interrupt_before breakpoint re-fires on every resume — run livelocks
Summary
A run interrupted at an interrupt_before breakpoint, when resumed, re-evaluates the same breakpoint condition against state that has not changed (the node still has not executed). The interrupt fires again → the run pauses again → resume → interrupt. The run can never pass the breakpoint: livelock.
Root cause
Breakpoint consumption is not persisted. Whether an interrupt_before for node N has already fired lives only in transient per-tick state (or is never tracked at all), so a resume re-enters the same path. Checkpoint metadata does not record consumed breakpoints, so resumeRun cannot skip them.
Reproduction
- Workflow: 2 chained task nodes
a → b, with interrupt_before: ["b"] (workflow-level or node-level breakpoint config).
- Start the run; it pauses before
b (run.interrupted fires).
- Resume the run (
POST /runs/{id}/resume).
- The engine re-evaluates the pre-
b breakpoint → interrupts again. Repeat forever; b never executes.
Expected behavior
A consumed breakpoint must be persisted (e.g. in the checkpoint metadata as consumed_breakpoints for the run) and skipped on subsequent evaluations — resume proceeds into b and the run completes. The run.interrupted callback/event should fire exactly once per breakpoint.
Bug:
interrupt_beforebreakpoint re-fires on every resume — run livelocksSummary
A run interrupted at an
interrupt_beforebreakpoint, when resumed, re-evaluates the same breakpoint condition against state that has not changed (the node still has not executed). The interrupt fires again → the run pauses again → resume → interrupt. The run can never pass the breakpoint: livelock.Root cause
Breakpoint consumption is not persisted. Whether an
interrupt_beforefor node N has already fired lives only in transient per-tick state (or is never tracked at all), so a resume re-enters the same path. Checkpoint metadata does not record consumed breakpoints, soresumeRuncannot skip them.Reproduction
a → b, withinterrupt_before: ["b"](workflow-level or node-level breakpoint config).b(run.interruptedfires).POST /runs/{id}/resume).bbreakpoint → interrupts again. Repeat forever;bnever executes.Expected behavior
A consumed breakpoint must be persisted (e.g. in the checkpoint metadata as
consumed_breakpointsfor the run) and skipped on subsequent evaluations — resume proceeds intoband the run completes. Therun.interruptedcallback/event should fire exactly once per breakpoint.