Add /doctor and automatic health checks for stale goals, leaked processes, shared worktrees, and runaway turns

[yxlyx]

Problem

The focused fixes in #318, #198, #199, #320, #217/#306, and #266 are necessary, but this incident also exposed a cross-cutting detection gap: Graff had no read-only health check that could tell the user or controller that its persisted state and the live machine disagreed.

Examples from one session:

• a completed old /goal was still injected into unrelated user turns;
• the old goal was paired with a checklist created by a different task;
• a trajectory reported server cleanup while an owned Next.js tree remained alive for three days;
• the server launcher had exited (PPID 1) while descendants retained Graff's root process group;
• 20 stale Graff roots accumulated across PTYs;
• multiple root sessions used the same uncommitted worktree;
• child/judge expansion reached thousands of hidden rounds;
• foreground bash calls blocked for hours without a visible deadline.

Each subsystem needs its own prevention fix, but a common invariant/health layer should detect recurrence early and provide attributable evidence.

Proposal

Add both:

1. a read-only graff doctor / /doctor command; and
2. lightweight automatic checks at startup, turn boundaries, normal completion, and session shutdown.

The checker should use harness-owned state rather than asking the model to infer health from conversation text.

Required checks

Goal and todo invariants

• GOAL_TODO_EPOCH_MISMATCH: active goal and todos have different ownership epochs.
• ACTIVE_GOAL_AFTER_SUCCESS: normal accepted completion left its goal active.
• COMPLETED_CHECKLIST_ACTIVE_GOAL: all scoped todos are complete but steering will continue.
• GENERATED_STEERING_IN_HISTORY: generated goal/checklist steering is being persisted as literal user history.
• Show goal ID/epoch, status, age, todo owner, and why steering will or will not be appended.

These checks complement the state-model fix and regression tests in #318; they must not normalize mismatched state by silently rebinding unrelated todos.

Root-session and worktree invariants

• DUPLICATE_WORKTREE_OWNER: more than one live root session owns the same canonical Git worktree.
• STALE_SESSION_LEASE: lease exists but PID/start identity is dead or mismatched.
• IDLE_ROOT_SESSION: root has exceeded a configurable inactivity threshold.
• Show PID, process start identity, session/trace ID, cwd/worktree, goal status, last model/tool activity, and child count.

Run duplicate-owner preflight before the first model request, as proposed in #320.

Owned process/job invariants

Maintain an explicit job registry for tool-started long-running processes. Check:

• ORPHANED_OWNED_JOB: launcher exited or job was reparented while descendants remain.
• JOB_SHARES_ROOT_PGID: a background job cannot be safely group-terminated without killing Graff.
• LISTENER_AFTER_CLEANUP: a recorded port remains open after claimed cleanup.
• DESCENDANTS_AFTER_TOOL_EXIT: child tree remains after the owning tool is complete.
• HOT_IDLE_SERVER: owned server exceeds configurable CPU/RSS/open-file limits without meaningful activity.
• CLOSE_WAIT_ACCUMULATION: owned socket count/state exceeds threshold.

Show PID/PPID/PGID, process start identity, descendant count, port, age, CPU, RSS, open-file/socket counts, and owner tool/turn. This complements #198 and #199.

Trajectory and tool invariants

• Display and enforce the aggregate budget state from Governed runs (beyond codex): aggregate attempts, time, tokens, spend, and descendant tool budgets #217/Add budget and checkpoint guardrails for runaway review trajectories #306: root rounds, descendant rounds, judges, tool calls, elapsed time, tokens/context, retries, and remaining budget.
• BUDGET_NEAR_LIMIT and BUDGET_EXCEEDED must be harness decisions, not model prose.
• FOREGROUND_TOOL_NO_HEARTBEAT: warn when a foreground tool has no completion/progress beyond a short threshold.
• FOREGROUND_TOOL_DEADLINE_EXCEEDED: terminate/escalate according to Parallel bash calls can remain stuck after interruption and obscure progress #266 rather than waiting indefinitely.
• Detect retry storms and context growth that can keep stale steering alive.

Automatic check points

• Startup: stale leases/processes, duplicate worktree ownership, persisted goal/todo consistency.
• Before each model/tool dispatch: budget and terminal-state checks.
• During long tools: monotonic deadline and heartbeat/progress checks.
• Before attempt_completion is accepted: goal transition and owned-job/listener checks.
• Shutdown/interruption: terminate owned groups according to policy, then verify descendants/listeners are gone.
• Next startup after a crash: reconcile the durable job/session registry without trusting PID alone.

Output and remediation

• Human-readable table plus stable JSON output for support/automation.
• Stable check IDs, severity (info, warn, error), evidence, and a safe remediation command/action.
• Never kill a process merely because its PID matches a stale record; validate executable, process start identity, owner token, and PGID.
• Default /doctor is read-only. Mutating remediation requires explicit confirmation unless a narrowly scoped, predeclared owned-job cleanup policy applies.
• Do not log raw commands, secrets, or unrelated process details; retain privacy-safe fingerprints/categories.

Acceptance tests

Use injected clocks and fake process/session tables so tests are deterministic.

1. Completed goal A plus todo epoch B produces GOAL_TODO_EPOCH_MISMATCH and no stale steering.
2. Normal successful completion transitions goal A and a later unrelated message receives no goal block.
3. Second root in the same canonical worktree warns before model execution; separate worktrees do not.
4. Reused PID with a different start identity is never signaled.
5. Tool shell exits while a registered server descendant remains: checker reports ownership and cleanup removes/verifies the dedicated group.
6. Cleanup claim with an open listener fails the completion health check.
7. Child/workflow/judge usage is charged to the aggregate budget and dispatch stops at the limit.
8. Foreground tool with a fake elapsed time beyond its deadline is interrupted deterministically.
9. Healthy single-session runs produce no warnings and incur bounded overhead.

This issue is an observability/invariant umbrella, not a replacement for the linked subsystem fixes.

[justrach]

Partially addressed in v0.0.226 (PR #328) - leaving this open.

Landed: A first /doctor slice: a pure, unit-testable check runner with the goal/todo-epoch and stale-goal checks.

Still open: most of the listed check IDs - DUPLICATE_WORKTREE_OWNER, ORPHANED_OWNED_JOB, BUDGET_EXCEEDED and friends need a job registry that does not exist yet.