Repository navigation
🤖 tests: the bug bash defaults to Sonnet alone and refuses a budget below its estimate - #6191
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f074899636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f074899636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4e0c5f3a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e4e0c5f to
edcaa79
Compare
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
edcaa79 to
d50869f
Compare
coder#6195) ## Summary When the probe gets HTTP 401 or 403, the bug bash now refuses with exit 2 and `ANTHROPIC_API_KEY rejected by <host> (HTTP 401): check the key`. This works in every `BUGBASH_AI` mode and for both `make bug-bash` and `make mcp-apps-e2e`. Refs coder#5714. Rebased onto main after coder#6191 (U4) merged. ## Background The readiness review of coder#6190 found a gap: a mock run with a rejected key ignored the probe's 401 and started every job. Each explorer's first call then got 401. A non-200 answer keeps its full reservation (about $0.64 per call), so the jobs then failed with misleading budget refusals (402) instead of naming the key. ## Implementation - `launch.ts` exports `keyRejected(job, status, records)`. It returns the refusal text for 401 or 403, naming the host but never the URL path or the key. It also reads the upstream status from the probe's call records, so a 401 with an unreadable body (HTML from a gateway), which `probeApp` reports as 502, is caught too. - `run.ts` checks it right after the unreachable-upstream check, before the `auto` fallback to the mock, so it applies in every mode. - The MCP Apps probe in `launchJob` uses the same text. - The fake upstream gets `[fake:401]` and `[fake:403]` modes, in Anthropic's error shapes, and `[fake:401html]`. ## Validation - I wrote the tests first, and all 4 failed before the change. `run.test.ts` covers mock/401, auto/401 and real/403. `runner.test.ts` covers an MCP Apps job against a fake 401: it refuses before `docker run` and keeps the probe's record. - Dogfood against a loopback fake upstream answering 401 (no provider reached): `bun tests/bugbash/run.ts` in mock and auto, and the MCP Apps launch, each exited 2 with `ANTHROPIC_API_KEY rejected by 127.0.0.1:<port> (HTTP 401): check the key`. - `bun test ./tests/bugbash` (Bun 1.3.12) passed 279 tests, and `make static-check` passed. ## Risks Low. A 401 or 403 from the probe already refused in `real` and `auto` mode. Only the text changed there. In mock mode, a run that a broken key used to let start now refuses. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$24.76`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=24.76 -->
## Summary `docs/AGENTS.md` gets one "Run a Bug Bash" section. With it, a new user can run `make bug-bash` and `make mcp-apps-e2e` from the docs alone. It also fixes the stale lines that described the bug bash before the sandbox. Refs coder#5714. **Merge after coder#6195.** The runbook quotes its rejected-key message (`ANTHROPIC_API_KEY rejected by <host>`), which main does not have until coder#6195 lands. ## Background A fresh-user audit of the bug bash found 12 doc gaps. The docs gave no host requirements, no image pull, no defaults, no cost, no output paths and no way to rerun one charter. They also had stale statements: host Node.js and `E2E_NODE` for the sandboxed targets, an app AI described as always the mock, and terminals running on the host. coder#6189, coder#6190, coder#6191 and coder#6195 changed the behavior, and this section describes the final result. ## Implementation - **New section**, before "Bug Bash and Repro Tests": - Host: Linux on amd64 (or arm64 with amd64 emulation), non-root, a local rootful Docker socket, git, make and Bun 1.3.12, and the ~1.4 GB image pulled on first run. - Env: `ANTHROPIC_BASE_URL` with or without `/v1`, and `BUGBASH_BUDGET_USD`. - Defaults: 6 charters × Sonnet 5.5 = 6 jobs, parallel 8, 6 steps, medium effort. - Cost: $9 by default, $12 at high effort, $29 with Opus, $5 for `mcp-apps-e2e`. - The two probe refusals, with a note that a 403 can mean the key has no access to the model (coder#6094 item 13 tracks the message text). - Where `findings.md` and the logs go, how to rerun one charter, `BUGBASH_MODELS`, `BUGBASH_EFFORT` and `MCP_APPS_E2E_ARGS`, the exit codes, and the $0 dry runs. - **Vendored skill:** `.xum/skills/bug-bash/` is unchanged. The new section lists which of its statements AGENTS.md overrides. - **Stale lines:** fixed in "Agent E2E" (no host Node.js for the sandboxed targets), the last sentence of the real-mode bullet, and `tests/bugbash/charters.txt:3`. - **Makefile, comments only:** the `bug-bash` comment now describes the probed app AI. Both targets warn not to share a make command line with a build goal under `-j`, because the inner `$(MAKE)` would build it a second time at once. - **Regenerated:** `builtInSkillContent.generated.ts`, from `docs/AGENTS.md`. ## Validation I checked every number and message against the merged code and the measured runs. The budgets are `estimateRun`'s output: `make bug-bash BUGBASH_BUDGET_USD=5` printed `set BUGBASH_BUDGET_USD to at least $9`, and `BUGBASH_EFFORT=high` needs $12. `make static-check` passed. The Makefile diff has 0 non-comment lines. ## Follow-ups (deferred from review round 2, coder#6094 item 14) - Exit 4 also means a launcher or e2e internal error, not only incomplete evidence. - After a stop or a proxy fault, `findings.md` lists only the charters that started. ## Fixed from review round 3 The amd64 host requirement, the cached-image timing in the Makefile comment, the mock-only claims in the override list, and the priced-model limit of `BUGBASH_MODELS`. A runner refusal of a non-amd64 host before the pull is code, so a separate PR delivers it. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$30.61`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=30.61 -->
1 participant
Summary
make bug-bashnow runs Sonnet 5.5 alone by default: 6 charters, 6 jobs. Before any job starts, it prints its cost estimate and a recommended budget. It refuses with exit 2 andset BUGBASH_BUDGET_USD to at least $Xwhen the budget is lower. Opus stays opt-in throughBUGBASH_MODELS. Refs #5714.Background
The old default was Opus 5.5 plus Sonnet 5.5: 12 jobs with 8 at a time. An audit estimated it at about $8 at list price, with $5–8 of the budget held by reservations of calls in flight. So a $10 budget could refuse calls late in the run, and a $5 budget failed partway, with no warning beforehand. The coordinator chose Sonnet alone plus an up-front check, after a clean-context review.
Implementation
make bug-bashruns at--max-steps 6and medium effort. Those runs (🤖 tests: run the bug-bash app and browser in a container sandbox #5714, 🤖 tests: run make bug-bash in the bug-bash sandbox, with the mock app AI #6076, 🤖 tests: run the real app AI in the bug-bash sandbox, through each job's proxy #6079) cost $9.50 for 21 charters. The estimate scales linearly with--max-steps. A model with no measurement gets Sonnet's cost × its list output price ratio × 2, and the run labels it(not measured: a guess). Opus comes to $1.80 per charter.min(--parallel, jobs)× the largest bound the proxy can reserve for one call of the costliest model. That bound is Sonnet $0.96 or Opus $1.93, from 200k input tokens at the cache-write rate plus 16,384 output tokens.proxyPolicy.tsnow computes the per-call bound and this worst case with one sharedcallBound. The formula and the limits are unchanged.--parallel 2Validation
run.test.tscases: the default refuses $8 and accepts $9, starts 6 Sonnet jobs and sends no probe before the refusal. A lower--parallelneeds less headroom. Opus gets a labeled guess. Both cases fail on the old code.BUGBASH_BUDGET_USD=5 bun tests/bugbash/run.tsprinted the estimate andset BUGBASH_BUDGET_USD to at least $9 (it is $5).bun test ./tests/bugbash(Bun 1.3.12) passed: 259 tests.make static-checkpassed.Risks
Low. The default now finds fewer distinct bugs than Sonnet plus Opus, and the run header says how to opt in. The check can refuse a run that would have fit: the headroom assumes every in-flight call reserves the maximum. The measured highest reservation was $0.64 for Sonnet, against the $0.96 assumed.
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$5.68