Conversation
CHECK_TIMEOUT_S was a module constant with no override path, so every task's check command was killed at 60s regardless of what the task declared. A task with "timeout_s": 2400 whose check runs `dotnet build && dotnet test` was SIGTERM'd at exactly 60.0s and reported TIMEOUT — a false negative on the mechanism that decides PASS, and indistinguishable on the scoreboard from a worker that produced nothing. Reproduced on 71763e6: task timeout_s=2400, check `sleep 65 && echo ok` -> elapsed=60.0s timed_out=True rc=-15. Resolution order, highest first: 1. the task's "check_timeout_s" (new, optional) 2. config.toml "check_timeout_s" (new, optional, install-wide default) 3. DEFAULT_CHECK_TIMEOUT_S = 60 (unchanged) Every resolved value is clamped to MAX_CHECK_TIMEOUT_S = 3600 and validated at parse time, so the gate is still guaranteed to terminate. Backwards compatible: a manifest that does not opt in resolves to 60, exactly as before. CHECK_TIMEOUT_S is retained as an alias. Verifier() still constructs with no arguments. Retry behaviour is untouched — TIMEOUT remains a retryable verdict and max_attempts stays 2. The repo-feature kit now declares check_timeout_s: 1800, since its check shells out to a real build/test command; its README explains when a kit must do this. tests/test_check_timeout.py: 19 tests covering resolution precedence, parse-time validation, config loading, actual subprocess enforcement, accurate timeout reporting, and the unchanged retry contract. Plus an opt-in unscaled proof (RINGER_SLOW_TESTS=1) that a check surviving 65 real seconds now passes. Suite: 179 -> 199 tests. The 3 pre-existing failures on 71763e6 (test_design_reference x2, test_scoreboard_page x1) are unchanged.
Uncommitted locally since 2026-07-23. engines/opencode-sandboxed.sh: RINGER_EXTRA_WRITABLE (colon-separated absolute dirs) widens the writable set. The .NET SDK writes NuGet caches, ~/.dotnet and MSBuild node state outside the repo; when those writes are denied MSBuild does not error, it HANGS with no output — undiagnosable from inside a worker. Entries are passed as -D params like every other path and never interpolated into the profile text, so the rule-injection guarantee still holds. docs/MODEL-NOTES.md: codex run record for hospedo phase-5c. The single FAIL was the check's bug, not the model's — the gate grepped \b85\b while C# writes 85m, and no word boundary exists between a digit and m, so a correct implementation could never pass. Recorded so the scoreboard is not read as evidence against codex on code-feature. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run's cost was effectively invisible. `worker_tokens` records a single step rather than the sum across steps, so the figure Ringer reports — and every cost estimate built on it — reads far below the truth. Measured on real tasks the gap is 19x to 40x. That is not a rounding problem. A fix swarm that cost $41.71 reported as roughly $3, and was restarted sixteen times over two days by an operator who had no way to see otherwise. Its most expensive single restart spent $19.43 and failed 31 of 36 tasks, because the manifest asked every worker to write a deliverable to a path its sandbox forbids. Nothing noticed the identical failures, nothing capped the spend, and the run only stopped when the provider's own key limit ran out — which takes every concurrent run down with it and cannot tell a productive run from a looping one. Providers already report the price of each model call in the worker's JSON stream, so: - parse_step_costs() sums that per task. It is exact: no price catalog, no assumption about the prompt/completion split, and it stays correct when a provider discounts or caches. An engine that reports no cost yields None rather than zero, so plan-billed work is not silently counted as free. - `budget_usd` is a hard ceiling, checked the moment each worker exits and before its check runs, so a slow verification cannot overshoot it. - `abort_after_repeated_failures` stops a run when N tasks fail in a row with the same signature — the shape an impossible manifest makes. - lint refuses a spec that orders the worker to write an absolute path outside its task directory. An absolute path in expect_files stays legal when the check produces it, which is the fix-swarm export pattern, so only the spec naming the path is flagged. - the summary prints per-task and per-run USD alongside step counts. All three controls default to off; existing manifests are unaffected. Tests provoke each gate on purpose, including the two cases that must NOT fire, and parse_step_costs is verified against real worker logs where it reproduces the provider's figures exactly ($0.463 / $0.093 / $0.068). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
Committed as-is to protect 206 lines of working-tree notes before bringing the branch up to date with upstream. Content unreviewed and unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
The fork was 21 commits behind, which cost real time: `run --baseline` (execute every check against the unmodified tree) already existed upstream and was being done by hand, and `setup_error` -- "an ERROR verdict at 0.0s carries no diagnostics anywhere the operator looks" -- was diagnosed manually after hitting exactly that. All five ringer.py conflicts were additive: this side adds check_timeout_s, upstream adds max_attempts, redact_spec and the update config. Both kept. config.sample.toml and MODEL-NOTES.md likewise. Upstream's test suite then caught a real bug in the local configurable-check-timeout work: `timeout_s: int = DEFAULT_CHECK_TIMEOUT_S` binds at import time, so any later change to CHECK_TIMEOUT_S was silently ignored. Resolved at call time instead, which keeps the per-task override and makes the install-wide setting actually take effect. 274 tests, 273 pass. The one failure is the contributor audit, which needs the README '## Contributors' section to name Barry Faassen -- left for upstream to add when PR NateBJones-Projects#129 merges, since that list is theirs to manage. The repo was also a shallow clone, so that audit could not run at all until `git fetch --unshallow`; it now does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
Two gaps that made the previous commit's numbers unsteerable rather than merely incomplete. A run could not say what a requirement cost. Measured on a real estate, 175 of 178 product tasks were keyed `fix-L13` and similar, so 96% of the money could not be traced to the work item it served. TaskSpec gains an optional `ticket`, lint requires it for task types that change a product (a bakeoff or probe legitimately serves none, so the rule is scoped rather than universal), and the eval log now records ticket, cost and step count. The three shipped templates that demonstrate product work gained the field -- they were the first thing the new rule caught. Codex reports tokens but never a cost, so every task it runs shows as free. On the same estate that was 797 of 922 tasks, every code review among them, which made review look free when it was the expensive half. Engines gain an optional `token_scale` and `price_*_per_mtok`. The scale matters more than it looks: Codex reports THOUSANDS -- its "tokens used" runs 12-230 across 40 real tasks where OpenCode reports hundreds of thousands for comparable work -- so an unscaled estimate is off by a factor of a thousand. An estimate is stored in its own field, never added to measured cost, and always labelled. With no prices configured the run says plainly that those tasks carry no cost of any kind and names them, because an unknown cost that defaults to zero is how a bill becomes a surprise. 7 further tests, including the two cases that must NOT fire: a research or probe task needs no ticket, and an engine with no prices yields None rather than 0.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
The ticket rule shipped with `dotnet-fix` and `dotnet-feature` hard-coded
into the source. Those are one estate's task-type names. Baking them into a
shared tool is how a shared tool stops being shared, and it would have made
the finding either wrong or invisible for every other user.
The default is now only this project's own documented vocabulary -- code-fix
and code-feature. An estate that coins its own product task types lists them
in config:
ticketed_task_types = ["code-fix", "code-feature", "dotnet-fix"]
A malformed value is refused rather than quietly ignored, because a setting
that silently does nothing is worse than one that is absent.
Tested through the real config path rather than a hand-built AppConfig: a
setting nothing can reach proves nothing about portability.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
An adversarial review failed this branch on three counts. All three were right, and all three came from the author verifying their own work. P1 — `budget_usd` was not a ceiling under parallelism. It was checked only when a worker exited, so with max_parallel=4 a $6 budget could reach $24 without one check firing. Live costs are now re-read on a timer while workers run, which closes most of the gap. It does not close all of it and the documentation no longer pretends otherwise: a task's price is not knowable before it runs, so nothing can be reserved, and the residual is whatever parallel workers spend between polls. "Hard ceiling" was a lie with a number attached; it now says "as soon as the spend is visible". P1 — `abort_after_repeated_failures` counted ATTEMPTS. `_note_failure` fired inside the retry loop, so one task failing twice identically reached a limit of 2 on its own and could stop the whole run. It now fires once the task is finished, retries included, which is what both the option name and the docs always said. P2 — the budget ignored estimated spend entirely, making it useless for the one case it is most needed: an engine that reports no cost of its own. On one estate that was 797 of 922 tasks. Enforcement now weighs measured plus estimated while REPORTING still keeps them apart, because presenting a guess as a measurement is how a number stops being trustworthy. Also documented what the failure signature cannot do. It is exit code plus first printed line, one streak across parallel tasks: two unrelated checks that both open with "FAIL" share it. A circuit breaker, not a diagnosis. The tests deserved the review's sharpest finding: the "must not fire" cases passed against unfixed code, because a rule that does not exist never fires. Each negative now sits beside its positive in one test, so it fails both when the rule is missing and when it is too broad. Verified in both directions -- 4 of the new tests fail against the pre-fix tree, all 22 pass after. 276 tests, 275 pass; the remaining failure is the contributor audit, which needs full git history and fails identically on an untouched tree here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
…olicy Five findings from the second review. Four fixed, one declined with a reason. NEW, and the worst of them: a run stopped by its budget could exit ZERO. The stop killed the workers, but `run()` returned success whenever every runtime that finished happened to pass, without consulting `stop_reason`. A budget stop could therefore pass a CI gate silently -- the exact shape of failure this branch exists to prevent. It now returns 1 and says why. The manifest field comment still read "Hard ceiling" after the README had been corrected, which is worse than never having fixed it: the two now disagreed, and a reader believes whichever they find first. The lint docstring claimed a worker may write in its task directory "and its assigned temp dir", while the code exempts only the task directory. The temp path is assigned at run time and is not knowable at lint time, so the doc was narrowed to what the code actually reasons about rather than the code widened to a promise it cannot keep. `ticketed_task_types` now defaults to EMPTY. Shipping `code-fix` and `code-feature` enabled was still one estate's idea of product work handed to every installation as a new lint failure it never asked for. The reviewer was right that "configurable" does not excuse a default policy. One more negative-only test was paired with its positive, and the count in the previous message was wrong: 22 methods, not 21. DECLINED -- the failure signature is exit code plus first printed line, so two unrelated checks that both open with "FAIL" can collide. Making it precise means tracking per-check failure families, which defeats the purpose: an impossible manifest fails IDENTICALLY, and that sameness is the whole signal. A more specific signature would make every task unique and the breaker would never fire. It is documented as a circuit breaker rather than a diagnosis, and the cost of a false trip is a stopped run, not a wrong result. 277 tests, 276 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
…expose Six findings, five fixed and one previously-declined finding now implemented after the reviewer rejected the reasoning. It was right to. P1, and the sharpest catch of the three passes: the cost ESTIMATE was priced from `runtime.tokens` -- the single-step field this entire change exists to condemn. Every engine with no measured cost, which is the only kind the estimate serves, was therefore underpriced by the same 19x-40x factor, and its budget exposure could sit under the limit while real spend continued. parse_step_tokens() now sums the stream; the legacy field remains only as a fallback for harnesses that stream nothing. `budget_usd` accepted NaN and infinity. NaN is truthy so it started the watcher, and `spent < NaN` is false, so the run stopped immediately for no reason. Infinity was a ceiling nothing could reach. Both now refused. The test file put `if __name__ == "__main__"` in the middle: five of eight classes were defined after it, so running the file directly executed 15 of 28 tests and silently skipped every one added since. Moved to the end. Direct execution now runs all 28. The README still granted a worker its "assigned temp dir" while the code reasons only about the task directory -- the same doc-contradicts-code shape as the ceiling wording last pass, in the file I had already corrected once. PREVIOUSLY DECLINED, now fixed: the failure counter was one streak, so interleaved parallel failures (A,B,A,B) never reached a threshold even though A had failed twice identically, and completion order could make unrelated failures look consecutive. It is now a tally per signature, which answers the question actually being asked. The reviewer's objection was concrete and the original defence -- that sameness is the signal -- argued for per-signature counting rather than against it. Template ticket examples are neutral placeholders instead of one estate's `work#...` convention. 277 tests via discovery, 28 in this file directly; the four new ones fail against the pre-fix tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
Five findings fixed. The one that mattered most was not a bug but an absence: every test of the budget and the circuit breaker called a helper and compared numbers. Nothing started the watcher, stopped a run, killed a worker or exercised a retry -- which is exactly where the previous three passes found their defects. tests/test_enforcement_e2e.py runs the real loop with no model involved. An engine is a binary plus arguments, so a shell that prints step_finish JSON is a perfectly good worker: the runner reads cost from the log either way. Three cases -- a budget stop that must also make the run report failure, identical failures that must stop the manifest early, and a healthy run that must NOT be stopped. That third one is the point: without it the first two would pass against a runner that always aborts. The two enforcement tests fail against upstream and pass here. Provider cost is untrusted input and was summed unchecked. A NaN makes `spent < budget` false forever and disables the stop; a negative refunds exposure the run has really spent. Both are now skipped rather than poisoning the total. The manifest budget was already validated; the log was not. The README claimed the breaker counts failures "in a row" while the code now keeps a tally per signature. The tally is right -- workers finish in arbitrary order and a streak misses the repetition worth stopping on -- so the documentation was corrected to match, not the code. The estimate's fallback comment claimed the legacy token field is broken. For the harness that actually needs the fallback that is unproven: measured over 619 Codex logs there are zero step_finish events, and its "tokens used" median sits in the same range as OpenCode's fresh input for comparable work, so it reads as a task total rather than one step. The comment now says what is known, what is not, and that token_scale is the per-engine correction. Remaining estate-specific anecdotes removed from the shared README, sample config and templates. 285 tests, 284 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
…ate's ledger Three findings fixed, two declined with reasons that are checkable rather than preferences. Configured estimate prices rejected negatives but accepted NaN and infinity, while feeding the same exposure comparison the budget already validates. A NaN price makes every comparison false and stops a run for a nonsensical amount. Now finite-and-non-negative, with the refusals provoked in tests. The shared source still contained this estate's ledger -- exact dollar figures, task counts, `work#666`, `fix-L13`, `dotnet-fix`. The README, sample config and templates had been cleaned two passes ago; ringer.py had not, which is the same half-finished shape as the ceiling wording. The lessons stay, the numbers are now general. An upstream reader should not have to read someone else's invoice to understand a comment. The breaker's stop message now names the tasks it counted. The signature is deliberately coarse, so a stop can be a real repeated failure or a collision between checks that open with the same line -- and the reader could not previously tell which. DECLINED, with evidence rather than preference: - "The signature still conflates unrelated failures." Making it more specific defeats the control. An impossible manifest fails identically across tasks whose checks differ -- that is exactly the case this exists for -- so adding a check-command discriminator would make every task unique and the breaker would never fire. The collision is real, is documented, and now names the tasks so it is diagnosable. The cost of a false trip is a stopped run, not a wrong result. - "Lint rejects a valid deliverable in the assigned temp dir." That path is assigned when the worker starts and carries a random suffix, so it cannot be written into a manifest in advance. The rejected case cannot arise. 285 tests via discovery, 30 in the cost-control file directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
…ires Two additive conflicts in ringer.py: upstream adds engine_bin_diagnostics, this branch adds ticketed_task_types. Both kept. The README Contributors entry is not self-promotion but the project's own enforced rule -- "every community PR that lands in main is credited here, that's a project rule, enforced by a test" -- and test_contributors fails without it. 295 tests, all passing. This is the first fully green run on this branch: the contributor audit needed `git fetch --unshallow` before it could run at all, and then it was correct to fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
RINGER_DIR points every workspace at this checkout, so merging here is what actually gives nexo, aios, monorepo and sombra-audio a spend meter, a budget that stops a run, an abort on repeated identical failures, and a lint rule for deliverables a sandboxed worker cannot write. One conflict, in the timeout constants. Kept the configurable-timeout block (which already exports CHECK_TIMEOUT_S as a backwards-compatible alias) and added BUDGET_POLL_INTERVAL_S beside it, dropping upstream's plain re-declaration rather than shadowing the resolver. 315 tests, all passing. Reviewed adversarially over five passes before this merge. Two findings were declined with reasons recorded in the branch history, and one limitation is documented rather than fixed: budget_usd stops a run as soon as the spend is VISIBLE and is not a hard ceiling -- a task's price does not exist until it has run, so nothing can be reserved. Measured overshoot at 20 parallel lanes: median $0.09, worst observed $1.12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125QPzJL8qBQsayptsvuPgB
Every control that existed fired after the spend: a meter, a budget, an abort. This is the half that comes before. Baseline stops being a flag. `run --baseline` was already shipped and had never once been used here, which is what a flag people are supposed to remember is worth. Every `run` now baselines itself first, and refuses the dispatch on the two shapes that make a task unbuyable: a check that is already green (green now, green at the end, so it can never tell you the work happened) and a check that could not be executed at all. A check that FAILS baseline is the wanted result and dispatches normally -- that is the whole point of the phase, and refusing on it would refuse every honest manifest. The canary is a stop, not a smaller batch. A multi-task run releases its first task alone, judges it by its own executed check, and only then releases the rest; a bad verdict marks every remaining task SKIPPED without spawning. Measured both ways on the same night: a 3-ticket run that did this caught a design fault on its first task, and a 32-task run that did not lost a whole round to a fault its first task had already demonstrated. --canary-confirm adds a human on top, deliberately not by default -- a pause met on every run becomes a keypress people learn to hit. Both gates take a REASON to skip, not a bare flag, and print WAIVED (not proved, not verified). A blank reason is rejected rather than quietly re-enabling the gate the operator believed they had turned off. The whole verdict lands in the run record as a `preflight` block, so "was this checked?" is answerable later instead of remembered. Two existing tests now waive baseline explicitly: the budget e2e checks `exit 0` on purpose, and the workdir-escape test needs to reach the runtime guard the gate would otherwise pre-empt. Both keep testing what they are named for, and both refusals are pinned independently in the new file. Every new test was run against the unmodified tree first: all nine fail there. Demo keeps its parallel fan-out -- the canary is skipped for it with the reason recorded, since serialising the first of three workers would hide the thing the demo exists to show. Its baseline passes cleanly (3 fail, 0 pass, 0 error). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
…deliverable a real refusal Two findings from the lens review, both about this change. The record answered "was a canary configured?" but not "was it judged, and what did it say?" -- and only the second question tells you whether the batch was released on evidence. The verdict now lands in the run record at all four points where the canary is actually decided: released, held, waived, and the single-task auto-skip. It lives on the state writer rather than in Preflight, which stays frozen and describes only what was decided BEFORE dispatch. The unreachable deliverable had no demonstrated failure case. The test provoked the `error` outcome through an escaping task key, which is a different fault -- the ticket's named case is a deliverable no sandboxed worker can write, and nothing detected that at all. Baseline now probes declared absolute `expect_files` against the nearest existing ancestor and refuses when nothing could be created there. Only absolute paths are probed, on purpose: a relative deliverable lands in the scratch dir the harness makes and is always writable, so probing those would refuse nearly every honest manifest. That complement is pinned too. The third finding, a failure-counter docstring contradicting its implementation, is declined as out of scope: `failure_counts` is untouched by this change and belongs to the cost-control work. It arrived because the review kit staged the wrong diff -- see the PR thread. All three new assertions were run against the previous commit first and fail there. Suite: 327 pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
The lens is right that `unwritable_deliverables` cannot establish that a WORKER can write a path: it probes as the dispatcher, and a sandbox can deny what `os.access` here calls writable. Widening the code is not available -- proving it needs a spawned worker, and spawning nothing is exactly what makes baseline free enough to run before every dispatch. So the claim is narrowed to what the code does. Naming the layers while correcting it, because the boundary is the useful part: lint's `worker_unwritable_paths` reasons about sandbox SCOPE and is the one that matches the measured incident; baseline catches what no process could write at all; the canary buys whatever neither could know statically, once rather than once per task. The docstring, the error text and the README now each say which of the three they are. Suite: 327 pass, unchanged -- this narrows claims, not behavior. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
…pawning nothing The docstring promised "Spawn nothing" and the function does spawn things -- every check is a subprocess, and the worktree path launches git helpers. The guarantee that actually matters, and the one the refusal depends on, is that no worker starts: no model, no billable token. Claiming more invites a maintainer to assume there are no side effects at all, when a check can legitimately export files, which the fix-swarm pattern relies on. Inherited wording, but this function is rewritten here, so it is corrected here. Suite: 327 pass, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
The demo disabled the canary by command name. Nobody asked for it, and unlike a real waiver it printed nothing -- it was recorded in the run record and silent on the terminal, which is the difference between a decision and a default nobody can see. A gate with a third path that the tool takes on your behalf is the shape this whole change exists to remove. The justification was that the demo shows parallel fan-out and the canary serialises the first of its three workers. That is true and it is not worth a special case: a demo of a path real runs never take teaches the wrong behaviour, and the canary is now what a real run does. The demo still passes end to end -- its baseline is clean (3 fail, 0 pass, 0 error) and its first task writes the file its own check demands. If the three-at-once visual is wanted back it is one flag away, and that flag announces itself. Which is the design. Suite: 327 pass, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
Prove a task can pass before buying it
…atch This entry was written on 2026-09-10 and deliberately not committed: the file had uncommitted changes from a concurrent factory run at the time. That run has since landed, so it goes in now. The model entry is the small half. The two process lessons are the ones worth having: An aggregate scoreboard row cannot separate "the model cannot do this" from "the harness was broken". glm-5.2's code-fix row read 0.17 first-try over 63 tasks and was used to justify routing a 24-ticket batch elsewhere -- but all 63 came from one sibling run whose dominant failure was a single block of 64 identical missing_expect_files. That is a harness signature. The same model then went 3/3 on a different factory's pilot. Group failures by run_id and read the failure mode before routing on the number. And: deliverables land in the worktree, the CHECK exports them. A scout task declared its deliverable at an absolute path outside the sandbox; the log shows the model finding the right answer in four tool calls and then burning ~40 on write, cat >, dd, cp, python, xattr -c and touch against a path it was never allowed to touch. The sibling fix-swarm tasks were immune only because their check exported from the check side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
Record what nexo Run 1 taught, including the row that nearly cost a batch
work#1031's canary took runtimes[0] -- manifest order, with no notion of whether that task looks anything like the batch behind it. Measured the same day it shipped: a 32-task scout ran 31 tasks on one model and task 1 on a weaker one left over from an earlier audition. Task 1 failed, 31 tasks were skipped, and the model they would have used had passed the identical check first try minutes earlier. That direction is free and reversible, which is why it was survivable. The other direction is not: a canary EASIER than its batch passes and releases work that then fails once per task, reaching the expensive failure THROUGH the gate meant to prevent it. Selection is now by (engine, model) -- both, because either alone misleads: two tasks on one engine may run different models, and one model under a different engine is a different harness. The most common pair wins, ties break by manifest order, and a uniform batch therefore keeps its first task exactly as before. Being right is not enough if nobody can see it, so the run now says which task was chosen, what it was moved off, and how much of the batch it speaks for -- and the record carries the same under preflight.canary.selection. A batch with no majority still runs its canary (one task's spend is cheap) but prints that a FAIL there means "this task failed", not "the batch is broken". That distinction is the whole point: a hold nobody can explain is how a gate gets waived by reflex. Five new tests, each run against the pre-fix tree first, where all five fail: a minority first task, a uniform batch, dominance beating manifest order, the exact half boundary, and a fragmented batch that warns. Suite 327 -> 332. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
…tually happened
The lens is right that the tests did not exercise it. Every case
differentiated tasks by ENGINE, so the model half of the (engine, model)
pairing was unproven -- and the model half is the one the measured incident
used: one harness, two models, the weaker declared first. Pairing on engine
alone would pick the failing task there and hold the batch, because all its
tasks share an engine.
The gap was not an oversight in judgement so much as a dead end I did not
push past: the first attempt gave two tasks different models on an engine
whose args_template has no {model} placeholder, and ringer correctly refused
it -- "model is set but engine mock has no {model} placeholder, so it would
be silently ignored". I took the refusal as "cannot test this" and moved to
engines. It meant "use an engine that takes a model".
So there is now one: mockm, with {model} ahead of {spec} because the mock
worker reads argv[-1] as its spec, leaving the model arg inert to it. The
new test is the incident in miniature -- weak-model first and failing,
strong-model x3 behind it -- and it fails against the pre-fix tree.
Suite 332 -> 333.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S
The canary must resemble the batch it gates, not merely come first
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
CHECK_TIMEOUT_S = 60is a module constant referenced directly insideVerifier._run_check, with no parameter, no config key, and no manifest field. A task'stimeout_sgoverns only the worker subprocess — it has never applied to the check.So any check that legitimately runs longer than 60 seconds is killed, regardless of what the task declares. The
repo-featurekit is the clearest case:checks/check_repo_feature.pysetstimeout=1800on its ownsubprocess.runfor the build command, so the kit believes it has 30 minutes while the parent gives it 60 seconds.Reproduced on
71763e6:This matters more than a slow build. A killed check produces verdict
TIMEOUT, which is written toruns.jsonland aggregated into the per-(model, task_type)scoreboard. On that scoreboard a slow-but-correct build is indistinguishable from a worker that produced nothing — so the routing signal Ringer exists to produce is quietly wrong for any lane whose check compiles or runs a test suite.The fix
Resolution order, highest precedence first:
check_timeout_s(new, optional manifest field)check_timeout_sinconfig.toml(new, optional install-wide default)DEFAULT_CHECK_TIMEOUT_S= 60 (unchanged)Every resolved value is clamped to
MAX_CHECK_TIMEOUT_S= 3600 and validated at parse time, so the gate is still guaranteed to terminate — an unbounded check would be an unbounded run.The
repo-featurekit now declares"check_timeout_s": 1800, since its check shells out to a real build/test command. Its README explains when a kit must set this.Backward compatibility
CHECK_TIMEOUT_Sis retained as an alias for anything referencing it.Verifier()still constructs with no arguments.TIMEOUTremains a retryable verdict andmax_attemptsstays 2.Tests
New
tests/test_check_timeout.py, 20 tests covering resolution precedence, parse-time validation, config loading and rejection, actual subprocess enforcement, accurate timeout reporting (the message names the resolved budget, not the constant), and the unchanged retry contract. One test is an unscaled proof that a check surviving 65 real seconds now passes — opt-in viaRINGER_SLOW_TESTS=1so it does not cost a minute on every run.Suite goes 179 → 199. The three failures present on
71763e6(test_design_reference×2,test_scoreboard_page×1) are unchanged — they are HTML/design-token assertions unrelated to execution; one asserts a hardcoded"Generated July 6, 2026"date string.Run with: