Repository navigation
🤖 tests: run the real app AI in the bug-bash sandbox, through each job's proxy - #6079
Merged
Merged
Conversation
ThomasK33
added this pull request to stack #6070
October 10, 2026 18:49
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Member
Author
Paid bug bash with the real app AI (functional acceptance of B3)One This run also covers the final B2 code. B3 sits on B2's final head The run printed
Generated with |
…b's proxy - run.ts resolves BUGBASH_AI (auto, real, mock) once on the host. The probe is one max_tokens 1 call through its own proxy and the run's ledger (launch.ts probeApp), never straight to the provider. aiMode.ts rules: auto falls back to the mock only when the upstream is unavailable; a rejected key refuses. `[mock:...]` charters stay on the mock. - launch.ts: an explore job with a resolved real mode puts BUGBASH_APP_MODEL (priced, Anthropic only) on its proxy's allowlist and passes the mode, never a key. entry.ts runs a real mode only in a model-driven job. - aiMode.ts: inside a model-driven sandbox job, real mode is the forwarder URL with a placeholder key. - B1 review follow-ups: modelDrivenSandbox() also checks the launcher's boot ID and nonce; startApp.ts refuses to start a proxied sandbox job without the marker or without all three XUM_DISABLE_* switches; a call that outlives close() fails the job (exit 5) and leaves nothing. - docs/AGENTS.md: the bug-bash app AI line (builtInSkillContent regenerated). --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
ThomasK33
force-pushed
the
tests/5714-b3-real-app-ai
branch
from
October 10, 2026 19:03
3dc21a9 to
b1d077e
Compare
…lls; launcher wording - startApp.ts (real mode): the naming agent gets BUGBASH_APP_MODEL. Naming and the sidebar status tried Xum's built-in Haiku 5.5 first, which the job's proxy refuses (6 refused calls in the paid B3 run). A $0 check against the fake upstream: before, the first naming call asked for claude-haiku-5-5; after, for claude-haiku-4-5. - runner.test.ts: the stop that run.ts sends after another job's fault aborts this job's in-flight call, keeps its reservation and ends the job. - docs/AGENTS.md: on the host the make targets are the launchers; only a model-driven e2e process refuses there. builtInSkillContent regenerated. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 082cd2f164
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The probe's proxy used the 10-minute call deadline and never saw the run's stop, so a silent upstream could hold `make bug-bash` for ten minutes before any job, and a signal could not end it (Codex). probeApp() now takes the run's stop (it closes the probe's proxy, which aborts the call) and a 10 s deadline, aiMode.ts's probe timeout. run.ts exits with the signal after a stopped probe. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
Member
Author
Paid recheck after the naming fix (B3)One
Total paid spend on B3: $3.1493 + $1.0054 + $0.00145. Generated with |
This was referenced Oct 11, 2026
yermakoffivan
pushed a commit
to yermakoffivan/mux
that referenced
this pull request
Oct 11, 2026
…1) (coder#6082) ## Summary Runtime fixes from the readiness reviews of the bug-bash sandbox PRs, with regression tests and their docs. Exit 3 now means only an unknown container state, a SIGKILLed launcher leaves one job folder, `recover` never treats an owner it cannot see as dead, and one unreadable leftover no longer stops `recover`. This is PR D1 of the coder#5714 plan, stacked on B3 (coder#6079). D2 (the self-check, the CI job and the runner docs) follows. Refs coder#5714 coder#5930 coder#5932 ## Implementation 1. **Exit 3 → 6** (`launch.ts` `exitFor()`, `run.ts` `charterCode()`): a job whose own e2e exits 3 (its infrastructure error) now gives launcher exit 6. Exit 3 means only that a container's state is unknown after cleanup. The precedence is unchanged: unknown cleanup (3), then a stop (130 or 143), then a proxy fault (5). The job's own code stays in `JobOutcome.code` and the job log. No Makefile, CI or script caller tests for exit 3. 2. **The proxy's private socket folder** (`proxy.ts` `privateParent`): the inner HTTP socket's 0700 folder goes into the job folder. The job folder itself is not mounted; only `stage/` and `proxy/` inside it are. The container still reaches the HTTP server only through the front socket, and a SIGKILLed launcher leaves one job folder instead of a separate `xum-bugbash-proxy-*` folder in `$TMPDIR`. The app-AI probe keeps its own folder. 3. **hidepid** (`ownerState()`): when `/proc` is mounted with `hidepid`, another user's live launcher has no `/proc` entry either. A gone PID is now "dead" only when the mount shows every process. Under `hidepid`, or when the mount policy cannot be read, it is "cannot tell". A present entry still decides either way. 4. **recover** (`runner.ts` `listJobs()`, `launch.ts` `recover()`): a leftover that cannot be inspected comes back as an error entry instead of stopping the whole command. recover reports it, leaves it, handles the other leftovers, and exits 3. Uncertain ownership never authorizes a removal. 5. **Docs** (`docs/AGENTS.md`, regenerated `builtInSkillContent.generated.ts`, the `hostPause.ts` header): - The budget scope: `BUGBASH_BUDGET_USD` caps one launch for `make mcp-apps-e2e`, and the whole run (all jobs plus the probe) for `make bug-bash`, at list price. - The sandbox exit codes and the proxy record path. - The manual cleanup of leftovers that recover leaves, and of a SIGKILLed launcher's folders. ## Validation 245 tests pass in `tests/bugbash/`. Each change has a test that fails with the change reverted: 1. The remap: `exitFor` and `run.ts` with a job exit of 3, an unknown cleanup, a stop and a proxy fault. 2. The socket folder: the fake job lists its job folder (`group passwd proxy stage xum-bugbash-proxy-X`). 3. hidepid: `hidepid=2`, `invisible`, `1`, an unreadable or missing mount line, and `hidepid=0`. 4. recover: two dead leftovers, the first not inspectable. The second is removed, the first stays, and recover exits 3. ## Risks Test-only code. The exit-code change is visible to anyone who scripted against exit 3 of the sandbox launcher. Nothing in the repo does. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$140.84`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=140.84 -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets
make bug-bashrun the real app AI inside the bug-bash sandbox. The app reachesBUGBASH_APP_MODEL(default Haiku 4.5) only through its job's provider proxy, the app-AI probe goes through the run's budget, and no provider key enters any container. This is PR B3 of the #5714 plan, stacked on B2 (#6076).Refs #5714
Background
B2 ran
make bug-bashin the sandbox with the mock app AI only. The real app AI needs the app's own provider calls to go through the same proxy and budget as the explorer's. B3 also takes three follow-ups from the B1 readiness review: a stronger sandbox check, a startup guard for the kill switches, and a test for a call that outlives the proxy's close.Implementation
run.tsresolvesBUGBASH_AI(auto,real,mock) once on the host. The probe is onemax_tokens: 1call through a proxy of its own and the run'sLedger(launch.tsprobeApp), so it never goes straight to the provider and counts against the budget. The aiMode.ts rules stay:autofalls back to the mock only when the upstream is unavailable, a rejected key refuses, andrealnever falls back. Charters that name a[mock:...]prompt stay on the mock. A mode resolved elsewhere (BUGBASH_AI_RESOLVED) refuses.launch.ts: an explore job with a resolved real mode puts the app model on its proxy's allowlist next to the explorer model, and passes only the mode and the model name. The app model must be priced and Anthropic. Repro jobs and the MCP Apps suite keep the mock.aiMode.ts: inside a model-driven sandbox job, the real mode is the job's forwarder URL with a placeholder key.entry.tsruns a real mode only in a model-driven job.inContainer.ts:modelDrivenSandbox()also checks the launcher's boot ID and its per-launch nonce, asentry.tsdoes, so another container with--init,--network noneand a socket at the mount path does not pass.startApp.ts(modelDrivenGuard): the app refuses to start when a sandbox job has a proxy socket but noBUGBASH_MODEL_DRIVENmarker, or when a model-driven job lacks any of the threeXUM_DISABLE_*switches. It is a guard behind thecommand.envpass-through, which once dropped the marker (B1).startApp.ts(real mode) sets the naming agent's model (agentAiDefaults.name_workspace.modelString) toBUGBASH_APP_MODELthrough the existing config API. Workspace naming and the sidebar status read that preference first, and otherwise start with Xum's built-in Haiku 5.5, which the job's proxy refuses (6 refused calls in the first paid run). No production default changes. If the Haiku 4.5 naming call fails, the built-in fallback still asks for Haiku 5.5, and the proxy refuses it before any upstream request. That costs nothing, and the allowlist stays as it is.Claims
probeApp()ledger is the run's. A test checks it is the same instance the jobs get. A probe proxy fault stops the run before any job (exit 5)docker runarg against the host keymodelDrivenGuardtests: a mounted socket without the marker refuses, a missing switch refuses, a repro job and the host start as beforeAll three new guards are test-proven: the test fails with the code reverted (nonce check, startup guard, app-model allowlist).
Validation
tests/bugbash/.--parallel 4.probe.proxy.jsonl), and the run printed "App AI: real (anthropic:claude-haiku-4-5 answered the probe)".XUM_DISABLE_AGENT_TOOLS=1,XUM_DISABLE_PROJECT_AUTOMATION=1andXUM_DISABLE_TERMINALS=1. 3 had noXUM_MOCK_AI, andmock-errors(a[mock:...]charter) had it.docker execproduced one Haiku call per real-mode job, settled through that job's proxy. The proxies refused nothing.max_tokens, messages, model, stream, system, streaming, noanthropic-beta, 0 tools. A1's allowlists hold.Risks
Test-only code. A real-mode bug bash now makes paid app-model calls, but only through the job's proxy, within the run's budget, and with the app's agent tools, terminals and project automation off.
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$112.03