feat(batch): --auto-restart supervisor and clawbench-batch-watch alerting

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 ### Added
+- Added `clawbench-batch --auto-restart N`, a supervisor that re-invokes an aborted batch with `--resume` into the same directory up to N times, and `clawbench-batch-watch`, which posts to a Slack/Discord webhook when a batch aborts, completes, or every N tasks. See [`docs/operations.md`](docs/operations.md).
 - Added `scripts/export_openeval.py`, an additive script exporting a batch's `rescore-summary.json` as an [EvalPort](https://github.com/adhabnr-ux/evalport) `ResultSet` Thanks to [@adhabnr-ux](https://github.com/adhabnr-ux).
 - Added a `--browser-runtime kernel` mode to the Harbor adapter that runs each task against one Kernel cloud browser, exposing only a credential-free CDP bridge to the agent, and finalizes the replay and deletes the browser during verification.
 

docs/cli.md

@@ -7,6 +7,7 @@ Every ClawBench command. From a PyPI install run them directly (`clawbench-run 
 | `clawbench` | Interactive TUI — guided model and test-case selection. Needs a TTY. |
 | `clawbench-run` | One task, one model (or human mode). |
 | `clawbench-batch` | A matrix of models × cases. |
+| `clawbench-batch-watch` | Post to a webhook when a batch aborts or finishes — see [`operations.md`](operations.md). |
 | `clawbench-rescore` | Re-judge trajectories you already have, without re-running agents. |
 | `clawbench-reproduce` | Download published traces for one leaderboard row and check you reproduce it. |
 | `clawbench-harbor-adapt` | Convert V2 into a Harbor dataset — see [`harbor.md`](harbor.md). |
@@ -74,6 +75,8 @@ Execution:
 | `--max-concurrent <n>` | 2 local, 1 Kernel/Browserbase | Parallel jobs |
 | `--stagger-delay <s>` | 15 | Minimum seconds between consecutive container starts (rolling start) |
 | `--resume <dir>` | — | Reuse a previous batch's output directory and skip finished runs |
+| `--auto-restart <n>` | `0` | Supervise the batch: re-invoke with `--resume` up to n times if it exits non-zero — see [`operations.md`](operations.md) |
+| `--auto-restart-delay <s>` | `30` | Seconds to wait before each re-invocation |
 | `--dry-run` | off | Print the job matrix without running anything |
 | `--output-dir <path>` | `test-output` | Base output directory |
 

docs/operations.md

@@ -0,0 +1,99 @@
+# Operating long batches
+
+A full V2 sweep is 8–20 hours per (model × harness). Over that span something
+outside the agent's control will go wrong — a container gets OOM-killed, a
+provider returns 502 for half a minute, a queue cap trips at task 75. This page
+covers what to do about it: how to resume, how to make resuming automatic, how
+to find out promptly, and what an abort does and does not cost you.
+
+## How to resume a run
+
+Every batch writes into one directory, `test-output/batch-<timestamp>/`, with
+one run directory per (case × model) and a `batch-logs/` folder beside them.
+That directory *is* the checkpoint. To continue an interrupted batch, re-run
+the same command with `--resume` pointing at it:
+
+```bash
+uv run clawbench-batch --models deepseek-v4-flash --cases-suite v2 --all-cases \
+  --harness hermes --resume test-output/batch-20260912-081500
+```
+
+Jobs the batch already finished are skipped; everything else runs. The rest of
+the command line must match the original — the batch directory records
+outcomes, not the flags that produced them.
+
+## Making resume automatic
+
+```bash
+uv run clawbench-batch --models deepseek-v4-flash --cases-suite v2 --all-cases \
+  --harness hermes --auto-restart 3
+```
+
+`--auto-restart N` runs the batch under a small supervisor. If the batch
+process exits non-zero — including being killed outright — the supervisor
+waits (`--auto-restart-delay`, default 30s) and re-invokes it with `--resume`
+into the same directory, up to N times. The supervisor holds no browser, no
+containers, and no model calls, which is exactly why it survives the failures
+the batch does not.
+
+Some things to know:
+
+- The batch directory is created *before* the first attempt, so every attempt
+  — including the first — runs as a resume into one place. You can also pass
+  `--resume` yourself to supervise an existing batch.
+- **Ctrl-C is never retried.** An interrupt stops the supervisor after the
+  current attempt winds down; only failures restart.
+- The final exit status is the last attempt's: `0` only when an attempt
+  finished with every job in a terminal state.
+- It is single-host. If the machine itself goes away, so does the supervisor;
+  resume by hand when it comes back.
+
+## Getting told when something happens
+
+```bash
+uv run clawbench-batch-watch test-output/batch-20260912-081500 \
+  --pid <batch pid> --heartbeat-every 10
+```
+
+`clawbench-batch-watch` polls a batch directory and posts to a webhook when
+the batch **aborts** (with the last task, an error tail from the newest log, and
+elapsed time), when it **completes** (with pass/total and wall-clock), and
+optionally a **heartbeat** every N completed tasks. It reads only artifacts the
+batch already writes, so it needs nothing from the batch process.
+
+Abort is detected two ways: if you pass `--pid`, the batch process exiting
+without a `batch-summary.json` is an abort; either way, `--stall-timeout`
+(default 45 minutes) with no new artifacts is treated as one.
+
+Configure routing once per operator in `~/.config/clawbench/notify.toml`:
+
+```toml
+webhook_url = "https://hooks.slack.com/services/…"   # or a Discord webhook
+interval_s = 60
+heartbeat_every = 10
+stall_timeout_s = 2700
+```
+
+Flags override the file. With no webhook configured the watcher still prints
+each notification, which is enough for a terminal or a log.
+
+## What survives an abort and what does not
+
+**Survives.** Every run directory that reached `run-meta.json` — its trace
+bundle, its interception result, its judge verdict if the judge ran. Those are
+what `--resume` skips.
+
+**Does not survive.** The task that was in flight when the process died. Its
+run directory may exist with a partial recording and no `run-meta.json`; on
+resume that task is re-run from scratch and the partial output is discarded.
+There is no action-level checkpointing inside a task, and there is no plan for
+one — a task is the unit of work.
+
+**Is not restored.** Batch-level bookkeeping from the aborted attempt.
+`batch-summary.json` is written at the end of an attempt, so an aborted attempt
+leaves none; the summary you get is the final attempt's, and jobs it skipped
+because they were already done appear there as `skipped`. Use
+`clawbench-rescore` on the batch directory for a full accounting across
+attempts.
+
+Related: [`docs/cli.md`](cli.md) · [`eval/scoring.md`](../eval/scoring.md)

pyproject.toml

@@ -26,6 +26,7 @@ Paper = "https://arxiv.org/abs/2604.08523"
 clawbench = "clawbench.tui:main"
 clawbench-run = "clawbench.runner.run:main"
 clawbench-batch = "clawbench.runner.batch:main"
+clawbench-batch-watch = "clawbench.runner.watch:main"
 clawbench-rescore = "clawbench.eval.rescore:main"
 clawbench-analyze = "clawbench.eval.analyze:main"
 clawbench-reproduce = "clawbench.eval.reproduce:main"

src/clawbench/runner/batch.py

@@ -949,6 +949,24 @@ def main() -> None:
             "any (case x model) job whose batch-logs/<case>-<model>.log already exists."
         ),
     )
+    p.add_argument(
+        "--auto-restart",
+        type=int,
+        default=0,
+        metavar="N",
+        help=(
+            "Supervise the batch: if it exits non-zero (OOM-killed container, "
+            "provider outage, queue cap), wait and re-invoke it with --resume up "
+            "to N times. Ctrl-C is never retried."
+        ),
+    )
+    p.add_argument(
+        "--auto-restart-delay",
+        type=float,
+        default=30.0,
+        metavar="SECONDS",
+        help="Seconds to wait before each --auto-restart re-invocation (default: 30)",
+    )
     from clawbench.runner.run import DEFAULT_HARNESS, HARNESSES
 
     p.add_argument(
@@ -998,6 +1016,23 @@ def main() -> None:
         suite = args.cases_suite or DEFAULT_CASES_SUITE
         args.cases_dir = CASE_SUITES[suite]
 
+    if args.auto_restart > 0:
+        # The batch cannot retry its own death, so a supervisor re-invokes it.
+        from clawbench.runner.supervise import (
+            resolve_batch_dir,
+            run_supervised,
+            strip_supervisor_flags,
+        )
+
+        sys.exit(
+            run_supervised(
+                strip_supervisor_flags(sys.argv[1:]),
+                batch_dir=resolve_batch_dir(args.resume, args.output_dir),
+                max_restarts=args.auto_restart,
+                delay_s=args.auto_restart_delay,
+            )
+        )
+
     rc = asyncio.run(async_main(args))
     sys.exit(rc)
 

src/clawbench/runner/supervise.py

@@ -0,0 +1,136 @@
+"""Re-invoke an aborted batch until it finishes — `clawbench-batch --auto-restart`.
+
+A multi-hour batch dies for reasons that have nothing to do with the agent:
+Chromium balloons and the kernel OOM-kills the container, a provider returns
+502 for thirty seconds, the queue cap trips at task 75. The batch process is
+gone, so nothing inside it can retry. This runs *outside* it: a small
+supervisor that spawns `clawbench-batch` as a child, and when the child exits
+non-zero waits and spawns it again with `--resume` pointing at the same batch
+directory, up to a bounded number of times.
+
+The supervisor owns nothing heavy — no browser, no containers, no model
+calls — which is exactly why it survives the failures the batch does not.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+import time
+from collections.abc import Callable
+from datetime import datetime, timezone
+from pathlib import Path
+
+DEFAULT_RESTART_DELAY_S = 30.0
+BATCH_MODULE = "clawbench.runner.batch"
+
+# Exit status for "the operator interrupted us", mirroring the shell's 128+SIGINT.
+INTERRUPTED = 130
+
+Spawn = Callable[[list[str]], int]
+Sleep = Callable[[float], None]
+
+
+def _default_spawn(cmd: list[str]) -> int:
+    """Run the batch as a child and return its exit status.
+
+    A KeyboardInterrupt here means the operator hit Ctrl-C: the child shares
+    our process group and received it too, so wait for it to wind down rather
+    than orphaning its containers, then re-raise so the loop stops.
+    """
+    proc = subprocess.Popen(cmd)
+    try:
+        return proc.wait()
+    except KeyboardInterrupt:
+        try:
+            proc.wait(timeout=60)
+        except subprocess.TimeoutExpired:
+            proc.terminate()
+        raise
+
+
+def strip_supervisor_flags(argv: list[str]) -> list[str]:
+    """Remove the flags the supervisor consumes so the child does not recurse."""
+    out: list[str] = []
+    skip_next = False
+    for arg in argv:
+        if skip_next:
+            skip_next = False
+            continue
+        if arg in ("--auto-restart", "--auto-restart-delay", "--resume"):
+            skip_next = True
+            continue
+        if arg.startswith(("--auto-restart=", "--auto-restart-delay=", "--resume=")):
+            continue
+        out.append(arg)
+    return out
+
+
+def resolve_batch_dir(resume: str | None, output_dir: str | Path) -> Path:
+    """The one batch directory every attempt resumes into.
+
+    `--resume` carries over an existing batch; otherwise a fresh batch
+    directory is created here so that the *first* attempt already runs under
+    `--resume` and every later attempt lands in the same place.
+    """
+    if resume:
+        return Path(resume).resolve()
+    ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+    batch_dir = Path(output_dir).resolve() / f"batch-{ts}"
+    batch_dir.mkdir(parents=True, exist_ok=True)
+    return batch_dir
+
+
+def run_supervised(
+    child_argv: list[str],
+    *,
+    batch_dir: Path,
+    max_restarts: int,
+    delay_s: float = DEFAULT_RESTART_DELAY_S,
+    spawn: Spawn = _default_spawn,
+    sleep: Sleep = time.sleep,
+    log: Callable[[str], None] = print,
+) -> int:
+    """Run the batch, restarting on non-zero exit up to ``max_restarts`` times.
+
+    Returns the final child's exit status: 0 only when an attempt finished
+    with every job in a terminal state. A child that was interrupted by the
+    operator is not restarted.
+    """
+    attempts = max_restarts + 1
+    cmd = [
+        sys.executable,
+        "-m",
+        BATCH_MODULE,
+        *child_argv,
+        "--resume",
+        str(batch_dir),
+    ]
+    rc = 1
+    for attempt in range(1, attempts + 1):
+        log(
+            f"[SUPERVISOR] attempt {attempt}/{attempts}: "
+            f"clawbench-batch --resume {batch_dir}"
+        )
+        started = time.monotonic()
+        try:
+            rc = spawn(cmd)
+        except KeyboardInterrupt:
+            log("[SUPERVISOR] interrupted by operator; not restarting")
+            return INTERRUPTED
+        elapsed = time.monotonic() - started
+        if rc == 0:
+            log(f"[SUPERVISOR] batch finished cleanly on attempt {attempt}")
+            return 0
+        log(
+            f"[SUPERVISOR] attempt {attempt} exited with status {rc} "
+            f"after {elapsed / 60:.1f} min"
+        )
+        if attempt < attempts:
+            log(f"[SUPERVISOR] restarting in {delay_s:.0f}s")
+            sleep(delay_s)
+    log(
+        f"[SUPERVISOR] giving up after {attempts} attempt(s); "
+        f"resume manually with: clawbench-batch --resume {batch_dir} ..."
+    )
+    return rc