ask: the spec never asked for answer.md - #85
Open
robreyreynolds wants to merge 1 commit into
Open
Conversation
The check on every ask run is `test -s answer.md`, but the packet prefix told the worker to answer directly and return only the answer — nothing in the spec the worker sees mentions answer.md at all. A real engine that follows that spec prints the answer to stdout and fails the check with a complete, correct answer sitting in worker.log. Reproduced live with a containerized Codex worker: rc=0, full answer in the log, verdict FAIL on a file the worker was never asked to write. The fake engines in test_ask_command could not catch this: they write answer.md unconditionally, whatever the spec says. That is why every ask test passed while every real ask failed. Fix: the prefix now opens with the output contract — write only the answer, in plain English, to a new file answer.md in the current working directory; stdout does not pass the check. The wording is deliberately compact because the prefix is load-bearing twice over: the eval log truncates the spec field, so a longer instruction pushes the request text out of the visible record and breaks test_default_keeps_request_visible_and_run_is_watched; and prefix bytes count against max_packet_bytes, so verbosity evicts top-ranked source passages and breaks test_hook_request_prefers_the_start_of_a_long_script. Two longer drafts failed exactly those two tests before this wording passed both. New test guards the contract at the packet: the instruction section of packet.text must name answer.md. Verified it catches the defect: run against the pre-fix tree, it fails on exactly that assertion; with the fix, the full suite matches the pre-fix baseline on this host. After the fix, a live ask through the same containerized Codex worker passes: answer.md written, check green, answer grounded in the source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every
askrun's check istest -s answer.md, but the packet prefix told the worker to answer directly and "return only the answer" — nothing the worker sees mentionsanswer.md. A real engine that follows the spec prints to stdout and fails the check with a complete, correct answer sitting inworker.log. Reproduced live with a containerized Codex worker: rc=0, full answer in the log, verdict FAIL on a file it was never asked to write.The fake engines in
test_ask_commandcan't catch this — they writeanswer.mdunconditionally, whatever the spec says. That's why every ask test passed while every real ask failed.Fix: the prefix now opens with the output contract: write only the answer, in plain English, to a new file
answer.mdin the current working directory; stdout does not pass the check. The wording is deliberately compact because the prefix is load-bearing twice: the eval log truncates the spec field (a longer instruction pushes the request text out of the visible record, breakingtest_default_keeps_request_visible_and_run_is_watched), and prefix bytes count againstmax_packet_bytes(verbosity evicts top-ranked passages, breakingtest_hook_request_prefers_the_start_of_a_long_script). Two longer drafts failed exactly those tests before this wording passed both.New test guards the contract at the packet: the instruction section of
packet.textmust nameanswer.md. Run against the pre-fix tree it fails on exactly that assertion; with the fix, the full suite matches the pre-fix baseline on this host (one pre-existing, unrelatedtest_deliverablesfailure reproduces identically with and without this change). After the fix, a live ask through the same containerized Codex worker passes:answer.mdwritten, check green, answer grounded in the source.