Repository navigation
🤖 tests: run the MCP Apps suite in the bug-bash sandbox through the provider proxy - #6073
Conversation
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93cc4128c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
93cc412 to
1411ebb
Compare
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1411ebba24
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Paid MCP Apps runs (functional acceptance of B1)Two runs of
Generated with |
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…et removal Follow-ups from the coordinator's clean-context review of the proxy PR, landing in B1 where the proxy first runs: - Refusals cost the container nothing, so only the first 20 are logged in full. After that stats().refusedBy counts them per category: the reason up to its first quoted (container-chosen) name, so the keys stay the fixed reason texts. - A message_delta that lowers or nulls a usage count of message_start keeps the reservation instead of settling on the lower number. - close() waits for both listeners to close, and a test checks that no socket file is left in the job folder. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$52.06`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=52.06 -->
…rovider proxy PR B1 of the #5714 sandbox plan (MCP Apps only; `e2e explore` and the real app AI are B2). - launch.ts: a second job kind, `e2e run --config e2e.mcpapps.config.ts`. Before any docker command it needs BUGBASH_BUDGET_USD, ANTHROPIC_API_KEY and ANTHROPIC_BASE_URL and an anthropic: explorer model, and it refuses BUGBASH_SCENARIO. It starts the provider proxy on a socket in the job folder (0700), mounts that folder read-only at /repo/.sandbox-proxy, closes the proxy as a stop hook, writes one record per call to `<output>.proxy.jsonl` on the host, logs the spend, and exits 5 when a call cost more than its bound. - inContainer.ts: the sandbox check (marker, docker-init as PID 1, only loopback) and the proxy check; entry.ts forwards 127.0.0.1:4141 to the socket for a model-driven job. - mcpapps/hostPause.ts passes only in a model-driven sandbox; there the MCP Apps agents get BUGBASH_MODEL through the forwarder with a placeholder key. e2e.config.ts (repros) keeps no model. - startApp.ts: a model-driven job gets all three kill switches with the mock app AI too. - inputs.ts: the host's BUGBASH_MODEL no longer passes; the launcher sets it per kind. - Makefile: mcp-apps-e2e builds dist and runs the launcher. - fakeUpstream.ts: a command mode for the zero-cost transport check. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$52.06`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=52.06 -->
…ambiguous ps match - runner.ts: runJob() takes the docker-run flags and the command (image first) apart, and CALLER_FLAG checks only the flags, so the job's own arguments (e2e's `-g`) pass. A comment records that `--detach` and `--rm=false` are not refused, and when they must be. - runner.ts: the name-and-labels lookup returns unknown when more than one container matches, instead of removing the first. - Tests for both. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
e2e starts the app with only `command.env`, and e2e.config.ts's allowlist omits BUGBASH_MODEL_DRIVEN, so startApp.ts never saw it and left agent tools, terminals and project automation on in the sandboxed MCP Apps job (Codex P1). The MCP Apps config loads only in a model-driven sandbox job, so it sets the marker itself. Real-Docker check: the app server's environ in the container had no XUM_DISABLE_* before, and all three after. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…ed proxy close, run paths - launch.ts: refuse a BUGBASH_MODEL the proxy cannot price before any docker command (every call would refuse). A stop now outranks exit 5: the bound-miss line is still logged, and exit 3 (unknown cleanup) still outranks both. - proxy.ts: close() waits at most 5 s for the aborted calls and the listeners, inside the launcher's 10 s stop-hook bound, and rejects if a call is still open. Test: calls stuck before the upstream headers, mid-stream and on a client that never reads, plus an idle connection, all settle at once as `kept`. - Makefile: mcp-apps-e2e pins BUGBASH_AI_RESOLVED=mock too, and the output path carries the shell PID, so two runs in the same second do not share it. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
6744b1e to
501bed2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 501bed2e09
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The launcher's record callback threw inside the proxy's call handler, and close() swallowed it, so a job could exit with its own code while `<output>.proxy.jsonl` missed calls (Codex). The launcher now keeps the first write error (its errno code only) and reports it as a proxy fault: exit 5, with the same precedence as a bound miss. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
1 participant
Summary
Runs the MCP Apps e2e suite (
make mcp-apps-e2e) inside the bug-bash sandbox, with itsagent.actexplorer reaching the model only through the A2 provider proxy. The host pause stays for every other model-driven run, and on the host the suite still refuses. This is PR B1 of the #5714 plan, stacked on C2 (#6072).Refs #5714
Background
#5815 paused every model-driven bug-bash run on the host, because model output picks actions there. A1 (#6059) and A2 (#6065) added a dormant provider proxy with a policy and a spend ledger. C (#6069) and C2 (#6072) made container cleanup and crash recovery safe. B1 is the first PR that turns the proxy on, for the MCP Apps suite only.
make bug-bash(e2e explore) follows in B2.Implementation
sandbox/launch.tsaccepts a second job kind:e2e run --config e2e.mcpapps.config.ts. Before any docker command it refuses withoutBUGBASH_BUDGET_USD, withoutANTHROPIC_API_KEYplusANTHROPIC_BASE_URL, with a non-AnthropicBUGBASH_MODEL, and with anyBUGBASH_SCENARIO. It starts the proxy on<jobDir>/proxy/sock, registers its close as a stop hook, mounts the socket folder read-only at/repo/.sandbox-proxy, and logs oneproxy:summary line. A proxied call that costs more than its reserved bound fails the job with exit 5.sandbox/inContainer.tsholds the isolation guard thatentry.tsandmcpapps/hostPause.tsshare:BUGBASH_CONTAINER=1, docker-init as PID 1, onlylo, and (for model-driven jobs) a real socket at the mount. It also holds the 127.0.0.1:4141 TCP forwarder thatentry.tsstarts, because the AI SDK speaks TCP.e2e.mcpapps.config.tsgives the agentscreateAnthropic({ baseURL: <forwarder>, apiKey: placeholder })(BUGBASH_MODEL). The default model isanthropic:claude-sonnet-5-5.startApp.ts(appSwitches) turns off agent tools, terminals and project automation for every model-driven job, also with the mock app AI.BUGBASH_MODELleftPASS_ENV. The launcher sets it per job kind, so a repro job never gets an explorer model.message_deltalowers or nulls a usage count, and waits until its socket file is gone afterclose().runJob()now takes the docker-run flags and the command apart, so CALLER_FLAG checks only the flags (e2e's-gafter the image passes). Cleanup returnsunknownwhen more than one container matches the name and labels.Threat-model claims wired here
inContainer.tsguard used byentry.tsandmcpapps/hostPause.ts. Tests: a fake/proc+/sys+ socket root passes, the host and a regular file in place of the socket refuseLedgerper job with theBUGBASH_BUDGET_USDcap. The host probe of the real app AI moves to B2 with the real app AI--envvalue against the keyValidation
tests/bugbash/, including the failing-first tests above.34eb9e50ac). The paid run below covers the final head. The suite itself fails there by design, because the fake model drives no useful actions.Risks
Test-only code. The new runtime surface is the proxy socket on the host and the forwarder in the container. Both exist only during a
make mcp-apps-e2erun, which needs an explicit budget and key. The host pause still refuses every other model-driven run.Known gaps:
awaitonclose()in the launcher./tmpjob folder, its docker client folder and itsxum-bugbash-proxy-*folder stay behind.unknownexit 3.mcp-apps-e2eis paused. PR D updates the docs.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$78.21