fix: ask asks for its own deliverable; checks get a per-task timeout - #32
Merged
Conversation
`ask` builds a one-task manifest whose check demands answer.md and whose expect_files declares it, but the context packet only ever said "answer the request directly in plain English" — nothing named a destination. The spec reaches the worker verbatim, so every real-engine run failed on a deliverable the worker was never asked to produce. `ask` runs max_attempts=1 by design, so there was no retry to rescue the mismatch. Append an explicit file instruction to the spec and route the filename through one ANSWER_FILE_NAME constant so the spec, the check, the expect_files entry and the answer reader cannot drift apart again. The instruction lives in one_request_manifest, not build_context_packet, because the file contract belongs to the manifest that verifies it — packets built for other callers keep no deliverable assumption. Test asserts the contract rather than the wording: every file the check verifies must also be named in the spec. Matches upstream issue NateBJones-Projects#102 / PRs NateBJones-Projects#85 and NateBJones-Projects#103, which are unmerged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ded 60s The check kill timer read the module-level CHECK_TIMEOUT_S directly, so a gate that builds, installs, or runs a real test suite was killed at 60s and recorded as TIMEOUT. That verdict is indistinguishable from a model failure on the scoreboard, and there was no knob to raise it — the manifest could extend the worker's timeout_s but never the check's. Add check_timeout_s to TaskSpec (default unchanged at 60) and thread it through Verifier.verify. _run_check keeps its timeout optional and falls back to the module global when it is None, so existing direct callers and the test that monkeypatches that global are unaffected. `run --baseline` picks the per-task timer up for free: it verifies through the same path. Tests cover parse/default/validation, that a task can raise the limit above the global, and that a runaway check is still killed at the task's own value. Matches upstream PRs NateBJones-Projects#91 and NateBJones-Projects#74, which are unmerged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent defects found while auditing what upstream has open that we don't. Both are present on
maintoday; both have unmerged upstream PRs behind them.1.
askwas broken on every real engineone_request_manifestbuilds a check that demandsanswer.mdand anexpect_filesthat declares it — but the context packet only ever said "Answer the current request directly in plain English. Return only the answer." Nothing named a destination, and the spec reaches the worker verbatim. The worker answers to stdout; the check fails on a file it was never asked to write.askrunsmax_attempts=1by design, so there is no retry to absorb it.Demonstrated against pre-fix code:
The fix appends an explicit file instruction to the spec and routes the filename through a single
ANSWER_FILE_NAMEconstant, so the spec, the check, theexpect_filesentry and the answer reader can't drift apart again. The instruction sits inone_request_manifestrather thanbuild_context_packet— the file contract belongs to the manifest that verifies it, and packets built for other callers should carry no deliverable assumption.Upstream: issue NateBJones-Projects#102, PRs NateBJones-Projects#85 and NateBJones-Projects#103 (unmerged).
2. The check timeout was hard-coded at 60s
_run_checkreadCHECK_TIMEOUT_Sdirectly. A gate that builds, installs, or runs a real test suite got killed at 60s and recorded asTIMEOUT— a verdict indistinguishable from a model failure on the scoreboard, with no way to raise it. The manifest could extend the worker'stimeout_sbut never the check's.Adds
check_timeout_stoTaskSpec(default unchanged at 60), threaded throughVerifier.verify._run_checkkeeps its timeout optional and falls back to the module global whenNone, so existing direct callers — and the test that monkeypatches that global — are unaffected.run --baselinepicks it up for free, since it verifies through the same path.Upstream: PRs NateBJones-Projects#91 and NateBJones-Projects#74 (unmerged).
Verification
RINGER_NO_SELF_UPDATE=1 python3 -m unittest discover -s testsringer.pyreverted and the new tests kept, all three new tests failasktest asserts the contract (every file the check verifies is named in the spec), not the wording, so rephrasing the instruction won't silently un-cover the bug🤖 Generated with Claude Code