Skip to content

BUG: /tracker/* endpoints block for the duration of tick() network I/O #41

Description

@addadi

Bug: /tracker/* endpoints block for the duration of tick() network I/O

Summary

All tracker HTTP endpoints (/tracker/status, /tracker/tasks, /tracker/stats, /tracker/task/{id}) share TrackerState.mutex with the tick loop. tick() holds that mutex across synchronous network calls (heartbeat, stall reconcile, NullTickets claim/poll) and subprocess interaction — so whenever a tick is in flight, every /tracker/* request blocks until the tick completes. With a slow NullTickets or a long dispatch, that is easily >10s of API unavailability, repeatedly.

Root cause

src/tracker.zig:297:

fn tick(self: *Tracker) !void {
    ...
    self.state.mutex.lock();
    defer self.state.mutex.unlock();

    self.heartbeatAll(tick_alloc);   // HTTP per running task
    self.detectStalls(tick_alloc);
    self.driveRunningTasks(tick_alloc);
    self.reconcile(tick_alloc);
    self.pollAndClaim(tick_alloc);   // HTTP claim
    self.cleanCooldowns();
}

Handlers take the same mutex (src/tracker.zig:1152, src/tracker.zig:1407). The mutex is a coarse lock covering both state mutation and I/O.

Reproduction

  1. Configure the tracker against a NullTickets endpoint, get at least one task running.
  2. While a tick is processing (e.g. add artificial latency on the NullTickets side, or watch logs), curl /tracker/status in a loop.
  3. Observe multi-second latencies aligned with tick activity; during dispatch-heavy periods the endpoints effectively hang.

Expected behavior

Read endpoints should be served from a consistent snapshot published by the tick loop (swap-under-short-mutex), so request handling never waits on network I/O. The dispatch/claim path stays as-is.

Suggested direction

Publish an immutable TrackerSnapshot (running-task JSON, counts, per-task detail map) at the end of every tick (and once at startup) under a dedicated snapshot mutex held only for pointer swaps. Handlers copy the view from the snapshot, never touching state.mutex. Wire format unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions