Skip to content

fix: ask asks for its own deliverable; checks get a per-task timeout - #32

Merged
usjoh merged 2 commits into
mainfrom
fix/ask-answer-file-and-check-timeout
Aug 16, 2026
Merged

fix: ask asks for its own deliverable; checks get a per-task timeout#32
usjoh merged 2 commits into
mainfrom
fix/ask-answer-file-and-check-timeout

Conversation

@usjoh

@usjoh usjoh commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Two independent defects found while auditing what upstream has open that we don't. Both are present on main today; both have unmerged upstream PRs behind them.

1. ask was broken on every real engine

one_request_manifest builds a check that demands answer.md and an expect_files that declares it — but the context packet only ever said "Answer the current request directly in plain English. Return only the answer." Nothing named a destination, and the spec reaches the worker verbatim. The worker answers to stdout; the check fails on a file it was never asked to write.

ask runs max_attempts=1 by design, so there is no retry to absorb it.

Demonstrated against pre-fix code:

UNFIXED CODE:
  check verifies : ('answer.md',)
  spec mentions answer.md? False
  max_attempts   : 1

The fix appends an explicit file instruction to the spec and routes the filename through a single ANSWER_FILE_NAME constant, so the spec, the check, the expect_files entry and the answer reader can't drift apart again. The instruction sits in one_request_manifest rather than build_context_packet — the file contract belongs to the manifest that verifies it, and packets built for other callers should carry no deliverable assumption.

Upstream: issue NateBJones-Projects#102, PRs NateBJones-Projects#85 and NateBJones-Projects#103 (unmerged).

2. The check timeout was hard-coded at 60s

_run_check read CHECK_TIMEOUT_S directly. A gate that builds, installs, or runs a real test suite got killed at 60s and recorded as TIMEOUT — a verdict indistinguishable from a model failure on the scoreboard, with no way to raise it. The manifest could extend the worker's timeout_s but never the check's.

Adds check_timeout_s to TaskSpec (default unchanged at 60), threaded through Verifier.verify. _run_check keeps its timeout optional and falls back to the module global when None, so existing direct callers — and the test that monkeypatches that global — are unaffected. run --baseline picks it up for free, since it verifies through the same path.

Upstream: PRs NateBJones-Projects#91 and NateBJones-Projects#74 (unmerged).

Verification

  • Full suite green: 336 tests, RINGER_NO_SELF_UPDATE=1 python3 -m unittest discover -s tests
  • Prove-fail: with ringer.py reverted and the new tests kept, all three new tests fail
  • The ask test asserts the contract (every file the check verifies is named in the spec), not the wording, so rephrasing the instruction won't silently un-cover the bug

🤖 Generated with Claude Code

usjoh and others added 2 commits August 16, 2026 06:10
`ask` builds a one-task manifest whose check demands answer.md and whose
expect_files declares it, but the context packet only ever said "answer the
request directly in plain English" — nothing named a destination. The spec
reaches the worker verbatim, so every real-engine run failed on a deliverable
the worker was never asked to produce. `ask` runs max_attempts=1 by design,
so there was no retry to rescue the mismatch.

Append an explicit file instruction to the spec and route the filename
through one ANSWER_FILE_NAME constant so the spec, the check, the
expect_files entry and the answer reader cannot drift apart again. The
instruction lives in one_request_manifest, not build_context_packet, because
the file contract belongs to the manifest that verifies it — packets built
for other callers keep no deliverable assumption.

Test asserts the contract rather than the wording: every file the check
verifies must also be named in the spec.

Matches upstream issue NateBJones-Projects#102 / PRs NateBJones-Projects#85 and NateBJones-Projects#103, which are unmerged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ded 60s

The check kill timer read the module-level CHECK_TIMEOUT_S directly, so a
gate that builds, installs, or runs a real test suite was killed at 60s and
recorded as TIMEOUT. That verdict is indistinguishable from a model failure
on the scoreboard, and there was no knob to raise it — the manifest could
extend the worker's timeout_s but never the check's.

Add check_timeout_s to TaskSpec (default unchanged at 60) and thread it
through Verifier.verify. _run_check keeps its timeout optional and falls
back to the module global when it is None, so existing direct callers and
the test that monkeypatches that global are unaffected. `run --baseline`
picks the per-task timer up for free: it verifies through the same path.

Tests cover parse/default/validation, that a task can raise the limit above
the global, and that a runaway check is still killed at the task's own value.

Matches upstream PRs NateBJones-Projects#91 and NateBJones-Projects#74, which are unmerged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@usjoh
usjoh merged commit 5cf97be into main Aug 16, 2026
2 of 3 checks passed
@usjoh
usjoh deleted the fix/ask-answer-file-and-check-timeout branch August 16, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant