Skip to content

ask: the spec never asked for answer.md - #85

Open
robreyreynolds wants to merge 1 commit into
NateBJones-Projects:mainfrom
robreyreynolds:fix/ask-answer-md-contract
Open

ask: the spec never asked for answer.md#85
robreyreynolds wants to merge 1 commit into
NateBJones-Projects:mainfrom
robreyreynolds:fix/ask-answer-md-contract

Conversation

@robreyreynolds

Copy link
Copy Markdown

Every ask run's check is test -s answer.md, but the packet prefix told the worker to answer directly and "return only the answer" — nothing the worker sees mentions answer.md. A real engine that follows the spec prints to stdout and fails the check with a complete, correct answer sitting in worker.log. Reproduced live with a containerized Codex worker: rc=0, full answer in the log, verdict FAIL on a file it was never asked to write.

The fake engines in test_ask_command can't catch this — they write answer.md unconditionally, whatever the spec says. That's why every ask test passed while every real ask failed.

Fix: the prefix now opens with the output contract: write only the answer, in plain English, to a new file answer.md in the current working directory; stdout does not pass the check. The wording is deliberately compact because the prefix is load-bearing twice: the eval log truncates the spec field (a longer instruction pushes the request text out of the visible record, breaking test_default_keeps_request_visible_and_run_is_watched), and prefix bytes count against max_packet_bytes (verbosity evicts top-ranked passages, breaking test_hook_request_prefers_the_start_of_a_long_script). Two longer drafts failed exactly those tests before this wording passed both.

New test guards the contract at the packet: the instruction section of packet.text must name answer.md. Run against the pre-fix tree it fails on exactly that assertion; with the fix, the full suite matches the pre-fix baseline on this host (one pre-existing, unrelated test_deliverables failure reproduces identically with and without this change). After the fix, a live ask through the same containerized Codex worker passes: answer.md written, check green, answer grounded in the source.

The check on every ask run is `test -s answer.md`, but the packet prefix
told the worker to answer directly and return only the answer — nothing
in the spec the worker sees mentions answer.md at all. A real engine
that follows that spec prints the answer to stdout and fails the check
with a complete, correct answer sitting in worker.log. Reproduced live
with a containerized Codex worker: rc=0, full answer in the log,
verdict FAIL on a file the worker was never asked to write.

The fake engines in test_ask_command could not catch this: they write
answer.md unconditionally, whatever the spec says. That is why every
ask test passed while every real ask failed.

Fix: the prefix now opens with the output contract — write only the
answer, in plain English, to a new file answer.md in the current
working directory; stdout does not pass the check. The wording is
deliberately compact because the prefix is load-bearing twice over:
the eval log truncates the spec field, so a longer instruction pushes
the request text out of the visible record and breaks
test_default_keeps_request_visible_and_run_is_watched; and prefix
bytes count against max_packet_bytes, so verbosity evicts top-ranked
source passages and breaks
test_hook_request_prefers_the_start_of_a_long_script. Two longer
drafts failed exactly those two tests before this wording passed both.

New test guards the contract at the packet: the instruction section of
packet.text must name answer.md. Verified it catches the defect: run
against the pre-fix tree, it fails on exactly that assertion; with the
fix, the full suite matches the pre-fix baseline on this host.

After the fix, a live ask through the same containerized Codex worker
passes: answer.md written, check green, answer grounded in the source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant