Bug: /tracker/* endpoints block for the duration of tick() network I/O
Summary
All tracker HTTP endpoints (/tracker/status, /tracker/tasks, /tracker/stats, /tracker/task/{id}) share TrackerState.mutex with the tick loop. tick() holds that mutex across synchronous network calls (heartbeat, stall reconcile, NullTickets claim/poll) and subprocess interaction — so whenever a tick is in flight, every /tracker/* request blocks until the tick completes. With a slow NullTickets or a long dispatch, that is easily >10s of API unavailability, repeatedly.
Root cause
src/tracker.zig:297:
fn tick(self: *Tracker) !void {
...
self.state.mutex.lock();
defer self.state.mutex.unlock();
self.heartbeatAll(tick_alloc); // HTTP per running task
self.detectStalls(tick_alloc);
self.driveRunningTasks(tick_alloc);
self.reconcile(tick_alloc);
self.pollAndClaim(tick_alloc); // HTTP claim
self.cleanCooldowns();
}
Handlers take the same mutex (src/tracker.zig:1152, src/tracker.zig:1407). The mutex is a coarse lock covering both state mutation and I/O.
Reproduction
- Configure the tracker against a NullTickets endpoint, get at least one task running.
- While a tick is processing (e.g. add artificial latency on the NullTickets side, or watch logs),
curl /tracker/status in a loop.
- Observe multi-second latencies aligned with tick activity; during dispatch-heavy periods the endpoints effectively hang.
Expected behavior
Read endpoints should be served from a consistent snapshot published by the tick loop (swap-under-short-mutex), so request handling never waits on network I/O. The dispatch/claim path stays as-is.
Suggested direction
Publish an immutable TrackerSnapshot (running-task JSON, counts, per-task detail map) at the end of every tick (and once at startup) under a dedicated snapshot mutex held only for pointer swaps. Handlers copy the view from the snapshot, never touching state.mutex. Wire format unchanged.
Bug:
/tracker/*endpoints block for the duration of tick() network I/OSummary
All tracker HTTP endpoints (
/tracker/status,/tracker/tasks,/tracker/stats,/tracker/task/{id}) shareTrackerState.mutexwith the tick loop.tick()holds that mutex across synchronous network calls (heartbeat, stall reconcile, NullTickets claim/poll) and subprocess interaction — so whenever a tick is in flight, every/tracker/*request blocks until the tick completes. With a slow NullTickets or a long dispatch, that is easily >10s of API unavailability, repeatedly.Root cause
src/tracker.zig:297:Handlers take the same mutex (
src/tracker.zig:1152,src/tracker.zig:1407). The mutex is a coarse lock covering both state mutation and I/O.Reproduction
curl/tracker/statusin a loop.Expected behavior
Read endpoints should be served from a consistent snapshot published by the tick loop (swap-under-short-mutex), so request handling never waits on network I/O. The dispatch/claim path stays as-is.
Suggested direction
Publish an immutable
TrackerSnapshot(running-task JSON, counts, per-task detail map) at the end of every tick (and once at startup) under a dedicated snapshot mutex held only for pointer swaps. Handlers copy the view from the snapshot, never touchingstate.mutex. Wire format unchanged.