Skip to content

[Node A] Supervisor, State Machine & Reaper Protocol — Full 6-Month Track (M. Yadav) #1

Description

@mohityadav8

Owner: M. Yadav · Project Lead / Systems Architect
Hardware: RTX 3050 · 6GB VRAM (Phi-3-Mini 4-bit supervisor model)
Scope: Orchestration, state machine, convergence detection, Reaper Protocol, cluster networking, WebSocket event backend
Duration: 6 Phases / ~26 weeks
Labels: node-a, orchestration, epic, reaper-protocol


🎯 Epic Goal

Own the "brain" of HydraNet — the state machine that drives every agent run, the Reaper Protocol that detects and kills dead-end code-fix loops, and the gRPC/WebSocket networking layer that ties Node B (inference) and Node C (sandbox) into one coordinated mesh. This is the single feature (Reaper) that differentiates HydraNet from every other local coding agent — it must work convincingly before anything else matters.

📅 Timeline (6 Months)

gantt
    title Node A — M. Yadav — 6 Month Roadmap
    dateFormat  YYYY-MM-DD
    axisFormat  %b
    section Phase I - Single Node
    Monorepo + State Machine (5 states)   :p1, 2026-07-06, 21d
    section Phase II - Reaper Protocol
    Convergence Scoring + Reaper Hook      :p2, after p1, 21d
    section Phase III - AST Retrieval
    pgvector Graph Store + Query Layer     :p3, after p2, 21d
    section Phase IV - Three-Node Mesh
    gRPC Contracts + Supervisor Router     :p4, after p3, 21d
    section Phase V - War Room Dashboard
    WebSocket Event Server + Auth          :p5, after p4, 21d
    section Phase VI - Benchmark & Ship
    Hardening + Soak Test + Release        :p6, after p5, 21d
Loading

🧭 Definition of Done (Epic-level)

  • All 6 phase demo milestones below are recorded and linked in this issue
  • hydra fix runs end-to-end across a real 3-laptop cluster without manual intervention
  • Reaper Protocol fires correctly on seeded death-spiral bugs with < 5% false-positive rate
  • v0.1.0 tagged and pushed to AnthropicBots/HydraNet

🟦 PHASE I — Single-Node Agent Loop

Demo Milestone: CLI fixes a real Python bug, end-to-end, on one laptop inside Docker.

  • 01. Scaffold monorepouv workspaces, packages: orchestrator, inference, sandbox, dashboard, shared; strict pyproject.toml (no floating versions, ruff + mypy configured)
    • Acceptance: uv sync installs clean from a fresh clone; ruff check . and mypy . pass with zero errors on scaffold
  • 02. Build core 5-state machinePLAN → IMPLEMENT → TEST → REFLECT → DONE/ABORT using LangGraph
    • Acceptance: unit test asserts every legal transition; illegal transitions raise a typed exception, not a silent no-op
  • 03. Typer CLI entrypointhydra fix --repo PATH --issue "description"
    • Acceptance: --help auto-generates from type hints; missing --repo fails fast with a clear error, not a stack trace
  • 04. Agent tool set (Phase I scope)read_file, write_file, list_dir, run_shell
    • Acceptance: every tool call and result is logged with timestamp + duration; write_file refuses paths outside the mounted repo slice
  • 05. State transition router — maps tool outputs → next state
    • Acceptance: router is a pure function (state, tool_output) → next_state, unit-testable without any live model or sandbox
  • 06. RunLogger — append every state, tool call, output, timestamp to a JSONL file
    • Acceptance: a killed process mid-run still leaves a valid, replayable JSONL (flush after every line, not buffered)
  • 07. Hard execution budget — max N steps, max M tokens, abort with clear log entry on breach
    • Acceptance: budget breach produces a distinct ABORT_BUDGET state, not a generic crash
  • 08. Config systemhydra.config.json (model endpoints, timeouts, retry limits, budget)
    • Acceptance: invalid config fails validation at startup with a field-level error message, not at runtime mid-fix
  • 09. Integration test — seed a known bug in a toy repo, run the CLI, assert the file changed correctly
    • Acceptance: test runs in CI, no external network calls, uses a local toy repo fixture
  • 10. Phase I demo — record terminal showing CLI fixing a real bug in a real small repo
    • Acceptance: video uploaded, linked in this issue, includes final git diff on screen

🟥 PHASE II — The Reaper Protocol (core feature)

Demo Milestone: Side-by-side recording — baseline agent looping 20+ times vs. HydraNet backtracking after 3 and solving it.

  • 01. ConvergenceDetector interface — inputs: diff history, command history, stderr history → output: score 0–1
    • Acceptance: interface documented with a docstring example; implementation swappable behind the interface
  • 02. patch_similarity() — normalized Levenshtein edit distance between last two diffs (on changed line sets)
    • Acceptance: identical diffs → 1.0, completely unrelated diffs → ~0.0, unit tests cover both extremes
  • 03. command_repetition_rate() — sliding window over last k shell commands, compute repetition fraction
    • Acceptance: k configurable via hydra.config.json, defaults to 3 as per README spec
  • 04. error_signature_match() — hash dominant error type + file from stderr, compare across attempts
    • Acceptance: two different stack traces pointing to the same file+line hash identically
  • 05. Weighted convergence_score aggregation — configurable α, β, γ weights
    • Acceptance: α·patch_similarity + β·command_repetition_rate + γ·error_signature_match, weights sum to 1, all in config
  • 06. ReaperHook — fires when score > threshold (default 0.75), halts branch, emits REAPER_FIRED event
    • Acceptance: threshold configurable; event includes score, attempt number, and triggering diff
  • 07. BranchManager — creates a git branch per approach (hydra/attempt-001), tracks genealogy tree as JSON
    • Acceptance: genealogy JSON is queryable and matches the exact tree shown in README's genealogy example
  • 08. ReaperNotifier — publishes REAPER_FIRED to the WebSocket stream for the dashboard
    • Acceptance: event delivered within 200ms of the Reaper firing, verified in integration test
  • 09. SpiralSimulator test harness — feed the agent a bug with no solution at the file it's looking at; verify Reaper fires at the right threshold
    • Acceptance: harness is reusable for regression testing every future change to convergence scoring
  • 10. Phase II demo — side-by-side recording: baseline looping 20+ times vs. HydraNet backtracking after 3 attempts and solving it
    • Acceptance: both runs use the identical seeded bug and identical model; attempt counts and token spend both shown on screen

🟩 PHASE III — AST Graph Retrieval

Demo Milestone: One bug that flat search misses, graph walk finds — show the exact query path through the graph.

  • 01. Deploy pgvector on Node A Postgres 16 — write schema migration
    • Acceptance: migration is idempotent, runnable via hydra cluster init
  • 02. Design graph schemacode_nodes(id, file, type, name, signature, embedding), code_edges(src, dst, edge_type)
    • Acceptance: schema diagram committed to /docs/architecture.md; edge_type enum covers import, call, inherit
  • 03. IndexManager — on hydra fix, scan repo, parse every .py file, embed chunks, upsert into pgvector
    • Acceptance: re-running on an unchanged repo is a no-op (see incremental indexing, task 04)
  • 04. Incremental indexing — hash each file's content, only re-embed changed files since last run
    • Acceptance: benchmark shows re-index of a 1k-file repo with 1 changed file completes in < 2 seconds
  • 05. VectorCacheLayer — LRU cache on the 100 most-recently-retrieved code chunks
    • Acceptance: cache hit avoids a DB round-trip, verified via query count assertion in tests
  • 06. RetrievalAPI endpointretrieve(error_context, k=10) → list[CodeChunk], called by the orchestrator
    • Acceptance: endpoint returns results ranked, with each chunk's file path and hop-distance from the error
  • 07. RetrievalEval harness — 20 seeded bugs, measure recall (did root-cause file appear in top-10?)
    • Acceptance: recall number is written to a report file, not just printed to console
  • 08. Stress test — index a 10k-file open-source repo (e.g. Django), measure indexing time and query latency
    • Acceptance: numbers documented in /docs/model-selection.md or equivalent, with hardware specs noted
  • 09. IndexHealth CLIhydra index-status --repo PATH shows stale files, coverage %, last-indexed timestamp
    • Acceptance: command works without needing to run a full hydra fix first
  • 10. Phase III demo — one bug flat search misses, graph walk finds; show exact query path through the graph
    • Acceptance: side-by-side output of both retrieval methods on the same bug, recorded

🟨 PHASE IV — Three-Node Mesh

Demo Milestone: Phase I CLI command runs across all 3 physical laptops over WiFi.

  • 01. gRPC proto definitionsOrchestratorService, InferenceService, SandboxService; commit as v1 contract
    • Acceptance: .proto files reviewed and tagged v1 before any implementation starts against them
  • 02. SupervisorRouter — dispatch CodeTask → Node B, TestTask → Node C, based on task type
    • Acceptance: routing logic is table-driven, unit-testable without live gRPC connections
  • 03. HeartbeatMonitor — ping each worker every 5s, mark UNHEALTHY after 3 missed beats
    • Acceptance: integration test simulates a dropped worker and confirms detection within 15s
  • 04. Worker failover — Node B goes UNHEALTHY mid-task → emit NODE_LOST, pause execution, wait for reconnect
    • Acceptance: no partial/corrupted state written during a pause; run resumes cleanly on reconnect
  • 05. NodeRegistry — Node B/C announce via hydra join --host IP --role on startup
    • Acceptance: registry persists across a supervisor restart (not purely in-memory)
  • 06. ResourceMatrix — supervisor tracks Node B VRAM usage + Node C container count, routes on capacity
    • Acceptance: routing avoids dispatching to a node reporting > 90% resource utilization
  • 07. EventBroadcaster — all significant events (task dispatched, Reaper fired, test passed) → WebSocket stream
    • Acceptance: every event type from Phase V's schema is emitted at least once in an integration test
  • 08. ClusterHealthCLIhydra cluster status shows all connected nodes, load, last heartbeat
    • Acceptance: output matches the exact format shown in the README's "Getting Started" section
  • 09. Network load test — sustain a 2-hour run on a real repo, measure packet loss, latency jitter, recovery from WiFi dropout
    • Acceptance: results documented with a graph of latency over time
  • 10. Phase IV demo — three laptops on a table, one hydra fix command, all three terminals active, screen recorded
    • Acceptance: video shows all 3 terminals simultaneously and the final successful fix

🟪 PHASE V — War Room Dashboard

Demo Milestone: Live UI showing genealogy tree, node meters, state stream.

  • 01. WebSocket event server on Node A — one connection per client, one event stream per active run
    • Acceptance: server handles 2+ concurrent client connections without cross-talk between runs
  • 02. Full event schemaTASK_DISPATCHED, AGENT_THOUGHT, TOOL_CALLED, TOOL_RESULT, REAPER_FIRED, BRANCH_CREATED, TEST_PASSED, RUN_COMPLETE
    • Acceptance: schema versioned and documented in /packages/shared/types.py
  • 03. JWT auth for dashboard WebSockethydra dashboard-token generates required token for WS URL
    • Acceptance: connection without a valid token is rejected with a clear 401-equivalent close code
  • 04. RunHistoryStore — every completed run written to SQLite; endpoint lists past runs + serves event log
    • Acceptance: history survives a supervisor restart
  • 05. LogExportEndpointGET /api/runs/{id}/export returns full JSONL event log as downloadable file
    • Acceptance: exported file is byte-identical to the original RunLogger output
  • 06. Streaming compression — compress large log payloads before WebSocket send to avoid browser frame drops
    • Acceptance: measured frame time stays smooth on a 10k-event backlog replay
  • 07. RateThrottler — batch/debounce if events arrive faster than 60/sec before sending to client
    • Acceptance: client never receives more than 60 discrete WS messages/sec under load test
  • 08. FuzzerTest for WebSocket server — 10,000 random events/sec, assert no dropped events or crash
    • Acceptance: test runs in CI on a schedule (nightly), not just locally
  • 09. MultiClientBroadcast — two browsers connected simultaneously see the same live run without interference
    • Acceptance: manual + automated test with 2 concurrent WS clients on the same run ID
  • 10. Phase V demo — screen record War Room Mode on a live three-node run, genealogy tree growing as Reaper fires
    • Acceptance: video shows convergence gauge, genealogy tree, and node meters all updating live

🟧 PHASE VI — Benchmark, Harden, Ship

Demo Milestone: v0.1.0 tagged and released with published benchmark numbers.

  • 01. 48-hour soak test — across all three nodes on a real medium-sized repo; log and fix every crash
    • Acceptance: zero unhandled exceptions in the final soak run; a log of all bugs found + fixed is committed
  • 02. Dependency pinning + audit — pin pyproject.toml/package.json; run pip-audit and npm audit; fix critical CVEs
    • Acceptance: zero critical/high CVEs remain open at release time
  • 03. SetupVerifier CLIhydra doctor checks Docker, Ollama, Postgres, Node B/C connectivity, reports clearly
    • Acceptance: each check reports PASS/FAIL with an actionable fix suggestion, not just a boolean
  • 04. Security hardening doc — sandbox network isolation, API token storage, what data leaves the machine
    • Acceptance: doc explicitly states "nothing leaves the machine unless a cloud fallback is configured," verified against actual code paths
  • 05. Single-command installerinstall.sh, tested on Ubuntu 22.04, Debian 12, WSL2
    • Acceptance: fresh VM test on all 3 target environments passes without manual intervention
  • 06. hydra cluster init — first-time setup, generates pre-shared token, writes hydra.config.json on all three machines
    • Acceptance: re-running init on an already-initialized cluster is safe (idempotent, doesn't rotate token silently)
  • 07. hydra update command — pulls latest version, runs DB migrations if schema changed
    • Acceptance: migration failure rolls back cleanly, doesn't leave the DB in a half-migrated state
  • 08. Dependency audit — remove anything unused, freeze the rest
    • Acceptance: pyproject.toml and package.json diffs reviewed and documented in the PR description
  • 09. Final repo cleanup — remove API keys, temp files, debug branches, TODOs not filed as issues
    • Acceptance: git log and grep -r "TODO" both clean before tagging
  • 10. Ship — tag v0.1.0, push, publish to GitHub
    • Acceptance: release notes link back to this issue and to Phase VI benchmark results from Y. Jangra's track

🔗 Dependencies

  • Depends on Node B track (Y. Jangra) for InferenceClient/gRPC InferenceService before Phase IV routing can be tested live
  • Depends on Node C track (B. Kataria) for SandboxManager/gRPC SandboxService before Phase IV routing can be tested live
  • Phase V dashboard frontend work is owned by Y. Jangra; this track only owns the WebSocket backend

⚠️ Risks

  • Convergence scoring thresholds (α, β, γ, 0.75 cutoff) are guesses until validated against real seeded bugs — budget time in Phase II for tuning, not just implementation
  • 6GB VRAM on Node A is tight for Phi-3-Mini + KV cache during long contexts — monitor headroom continuously

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

help wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions