🤖 tests: container runner for the bug-bash sandbox by ThomasK33 · Pull Request #5802 · coder/xum

ThomasK33 · 2026-10-06T22:05:37Z

Summary

This PR adds the container runner of the bug-bash sandbox: tests/bugbash/sandbox/launch.ts, the container entry entry.ts, the image Dockerfile and unit tests. It is PR 2 of 6 in the approved plan for #5714, on top of #5800 (the export stream, merged). It launches containers only: there is no host fallback, and no make target calls it yet. Today's host repro path (make test-bugbash-repros and the other targets) stays exactly as it is on main. Product code changes: 0 lines.

Plan for PR 3 (the user's "pause" decision). PR 3 must enforce the pause at runtime on today's host path, before any model-driven process starts: e2e explore, any config whose tests use agent.*, and run.ts. A static "no agent.* in repros" test alone is not enforcement, so PR 3 adds the runtime refusal as well.

Refs #5714

Background

Today the bug-bash app (xum server --no-auth), its Chromium and the e2e explorer all run on the host. Many app routes run host commands: executeBash, terminals, stdio MCP servers, editor commands and plugin installs. Only the charter text and three XUM_DISABLE_* switches keep injected text away from them. This runner puts the whole e2e job into one disposable container.

Implementation

What the container gets (runInSandbox()):

--network none: only lo. No host loopback, no Docker bridge, no internet.
--user <uid>:<gid> --cap-drop ALL --security-opt no-new-privileges --read-only, with pids and memory limits (4 GiB, no swap), --init and --log-driver none.
tmpfs for /tmp, /home/bugbash and tests/bugbash/.e2e. No host home and no Docker socket.
Read-only --mount type=bind mounts only (a missing source fails instead of creating a folder):
- a staged copy of the git-listed src, tests/bugbash, tsconfig.json and package.json. Ignored files stay out. A symlink (also a symlinked folder on the way) or a special file stops the launch.
- dist/ and node_modules/.
- a generated /etc/passwd and /etc/group.
Env: containerEnv() builds it from fixed values plus 8 allowlisted BUGBASH_*/E2E_* names (not BUGBASH_AI_REASON: it can hold a provider URL). It throws on any name that ends in _API_KEY, _AUTH_TOKEN, _TOKEN, _BASE_URL, _SECRET or _PASSWORD in any case, also from a caller. The docker CLI gets no user client config (see below).

Output. The job's --output .e2e/<folder> comes back through the export stream of #5800 into the same host folder. outputDir() accepts exactly one such folder, with no . or .. segment. The host folder must not exist yet, and no folder on the way to it may be a symlink. No end frame gives exit 4, "evidence incomplete".

Preflight, before anything runs:

checkEndpoint() refuses non-Linux hosts, root, a non-unix-socket DOCKER_HOST/context, Docker Desktop and rootless Docker.
checkSameHost() runs a probe container that reads the boot id (same kernel) and a random nonce through a bind mount (same files). A daemon reached through a shared socket from another container fails this check.

Image and build trust. xum-bugbash-sandbox:<hash of the Dockerfile and the Playwright version of @e2e-dev/web>. It holds Node 22, bun, git and Chromium only. The runner builds it when that tag is missing. No code removes or prunes images. The build runs outside the job sandbox: on the daemon, with the default build network (apt and the Chromium download need it), and without the job's limits. So its inputs come only from reviewed harness source:

The Dockerfile as committed at HEAD (git show HEAD:tests/bugbash/sandbox/Dockerfile). A work-tree change refuses (exit 2): commit it first. So no model-written edit reaches the build.
No build context: docker build - with the Dockerfile on stdin sends no folder. No demo repo, no .e2e evidence and no file that a job wrote can reach the build.
One build arg, PLAYWRIGHT_CORE_VERSION: the version of the installed playwright-core of @e2e-dev/web (from bun.lock), checked to be a plain x.y.z version. No build arg comes from the env, and the private client config adds no proxy args.
The builder is the local daemon's default docker driver: the empty client config has no buildx builder selection.
Residual risk: the base images (node:22-slim, oven/bun:1.3.12-slim) are pinned by tag, not by digest.

Crash cleanup. The launcher holds the container's stdin. When it dies, entry.ts sees EOF and kills every process in the container, and --rm removes it. entry.ts first checks that it runs in the sandbox (marker env, docker-init as PID 1, lo only) before it calls kill(-1). On SIGINT/SIGTERM or the 30 min deadline, the launcher removes its container. Removal matches the container's name, its owner label AND this checkout's xum.bugbash.checkout label, never the name alone. Even when removal fails, the launcher closes the lifeline and kills its docker client, so the deadline always holds. entry.ts waits until every other process is gone before the export. The sweep for frozen containers comes in PR 5. Staging: the launcher handles SIGINT and SIGTERM from before it creates the job folder. It yields to the event loop after the synchronous staging, and right before the job container starts, so a signal during staging or the probe stops the job with exit 130 or 143. Only this job's folder is removed: the launcher creates it without recursive, so the folder is its own or the launch fails.

Docker client isolation (repair push 2). The docker CLI copies the proxies entries of its client config (~/.docker/config.json or DOCKER_CONFIG) into every build as build args and into every container as env (HTTP_PROXY, HTTPS_PROXY, NO_PROXY and the lowercase forms). A proxy URL can hold a user and a password. Before this fix, that path skipped containerEnv()'s credential check, and the job could export the values. This PR does not claim that a Dockerfile can read registry credentials: I did not reproduce that.

One docker context inspect, with only the user's DOCKER_HOST, DOCKER_CONTEXT, DOCKER_CONFIG, HOME and PATH, resolves the selected endpoint. It reads the context selection only, and starts no build and no container. Only a local unix:/// socket passes. A remote, ssh or relative endpoint is refused, never swapped for the default socket.
Every other docker command (info, image inspect, build, the probe, the job, ps, rm) runs with PATH, DOCKER_HOST = that endpoint and DOCKER_CONFIG = a fresh, empty private folder that lives until the launcher exits. No HOME, no other DOCKER_* value, no credential, TLS file or credential helper.

Proof, with a synthetic credential only (a fixture client config whose authenticated proxies entry holds a random marker; the user's real config was never read or copied):

Where the marker can appear	Old head `c5fb22dddd`	This fix
Build env (a `RUN env` line in a throwaway Dockerfile variant, exported by the job)	4 lines (`HTTP_PROXY`, `HTTPS_PROXY` and lowercase)	0, and no `*_PROXY` name
Runtime env of the job, exported	4 lines	0, and no `*_PROXY` name
Image `Config.Env`, image history, launcher log	0	0

Caveat: BuildKit leaves proxy build args out of its cache key. A layer that an older launcher built with a proxy config can still be reused from the build cache. My first proof run showed this (two variants that differed only by a comment). The proof that counts used a separate RUN line per variant.

Real CLI context selection (contexts created only in a fixture DOCKER_CONFIG): a local context runs the sandbox (mock-only repros 6/6, export complete). A tcp:// context is refused (exit 2), never swapped for the default socket. (Measured on the previous head, where auto still had a host fallback; this head has none.)

Unit tests: a fake docker CLI logs every call's env names, DOCKER_HOST, DOCKER_CONFIG and the files in it. With a synthetic marker config, DOCKER_CERT_PATH and DOCKER_TLS_VERIFY set: only the context call sees the user's config. Every later call (info, image, build, probe, job, ps, rm, ps) sees exactly DOCKER_CONFIG DOCKER_HOST PATH, one empty private folder and no marker, and that folder is gone after exit. Another context is passed on as DOCKER_HOST. tcp://, ssh://, a relative unix:// and a remote context are refused after the context call alone. Mutation check: passing the user's DOCKER_CONFIG, passing HOME, falling back to the default socket, accepting a relative socket, keeping the folder, and spawning the job with the ambient env: each mutant failed a test.

Exact-step runs only (scope reduction in review). The runner runs only exact-step repro runs. exactStepRefusal() is an allowlist: e2e run, --config e2e.config.ts exactly once, selection and output options only, no positional file, no --, no dash-led option value, no explore argument, the cwd must be tests/bugbash, and e2e.config.ts must be a regular file there (e2e resolves --config from its cwd). Everything else (explore, the MCP Apps config, the real app AI) exits 2 until the provider proxy lands (PR 4). A repro that took the agent fixture anyway finds no model: the container has no network and no provider key. e2e 0.17 reads no env var that picks a config or an agent.

Containers only (scope reduction 2, approved by the user). The runner has no host fallback, and BUGBASH_SANDBOX is gone. Without usable Docker (no daemon, a remote, ssh or relative endpoint, Docker Desktop, rootless), with a failed same-host probe or with the real app AI, it refuses with exit 2 and a message that names the reason.

Validation

All on this host (Docker 27.5.1, 32 cores), run by hand through the runner:

Check	Result
Probe in the container	uid 1000, CapEff/CapBnd 0, NoNewPrivs 1, seccomp 2. Interfaces: `lo` only. `/`, `src`, `node_modules` and `dist` read-only. Empty home. No `/home/coder`, no docker socket. `127.0.0.1:22` ECONNREFUSED, `172.17.0.1` and `1.1.1.1` ENETUNREACH, DNS EAI_AGAIN
Env names in the container	`BUGBASH_AI_RESOLVED BUGBASH_CONTAINER HOME HOSTNAME NODE_VERSION NPM_CONFIG_UPDATE_NOTIFIER PATH PLAYWRIGHT_BROWSERS_PATH PWD TMPDIR YARN_VERSION`, with a fake `ANTHROPIC_API_KEY` and `GH_TOKEN` set on the host
Planted symlink and FIFO in the output	not exported. The host folder holds only regular files
`explore`, or no Docker	refused, exit 2
`DOCKER_HOST=tcp://…`, or a `tcp://` context	refused, exit 2
Launcher SIGKILL with a running container	container gone after 130–152 ms
Mock repros, 6 + 26 tests	all pass in the sandbox, both exports complete
Final head `fd37da52c0`: mock-only repros through the real launcher	sandbox 6/6, export complete. No daemon (`DOCKER_HOST=unix:///nonexistent.sock`): refused, exit 2, no output folder. SIGTERM while the container runs: exit 143, no container, job folder or client folder left
Symlinked `.e2e/linkx` with `--output .e2e/linkx/out`	before the fix: 2 files written outside the checkout. Now: refused, 0 files
Fake docker whose `run` hangs and `ps`/`rm` fail, then SIGTERM	before the fix: the launcher still ran 6 s later. Now: exit 143 at once
Container with another checkout's label	removal leaves it running (real Docker)
Build context: `COPY . /ctx` in a Dockerfile on stdin, run from `tests/bugbash` (36 tracked files, 60 evidence files)	BuildKit transferred 2 bytes of context, and `/ctx` held 0 entries
A work-tree line appended to the Dockerfile	refused before any build, exit 2
`make test-bugbash-repros` wall time, warm image (measured with PR 3's wiring)	sandbox 198.0 s and 194.7 s, host 193.8 s
First image build	included in a 238 s first run. The image is 1.43 GB

Unit tests (launch.test.ts): the env allowlist with host secrets set, credential names from a caller (also lowercase), the output-folder rules, symlinked folders, signal exit codes, and the exact-step allowlist. One test runs the launcher with a fake docker and a fake e2e node for 18 refused argument lists (second --config, --config=<other>, path variants, a positional file, --, explore anywhere, --agent, cache switches) plus a wrong cwd, and checks that nothing starts. Containers only: the real app AI, a missing context and a failed probe each refuse (exit 2) with no job container. Build: the build call is exactly build --build-arg PLAYWRIGHT_CORE_VERSION=<x.y.z> -t xum-bugbash-sandbox:<hash> -, and its stdin equals the committed Dockerfile. Staging cleanup: SIGINT and SIGTERM during the staging and during the probe exit 130 and 143, remove this job's folder and the client folder, keep another job's folder and another checkout's folder, and start no job container. A mutation check removed each exact-step rule, and the check on each path, one at a time: each mutant failed a test. A mutation check removed the credential check, the .. check, the single---output check and the allowlist, one at a time. Each mutant failed a test. make static-check passes, including hadolint and test routing. Mutation check for this push: no event-loop yield, no staging checkpoint, no final checkpoint, a work-tree Dockerfile, an extra env build arg, a build context, accepting the real app AI, ignoring a failed probe: each mutant failed a test.

Dogfood recording (from an earlier head): a 1x replay of a live asciinema recording of the first four rows above.

Risks

Low for the product: test tooling only, and nothing calls it until PR 3. Size: 1069 changed lines (runner 523, tests 437, entry 90, Dockerfile 19). Review fixes grew it from 493. The coordinator approved 758 lines as a scope-reduction exception, and the user's waivers covered the Docker client isolation push and this containers-only push. The kernel and the Docker daemon are trusted. A container escape is out of scope.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $73.62

chatgpt-codex-connector · 2026-10-06T22:05:46Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T07:12:05.375248Z	`fd37da5`	New commits
🔒 Security Review	✅ Completed 2026-10-07T07:07:46.027640Z	`fd37da5`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

ThomasK33 · 2026-10-06T22:06:08Z

Dogfood evidence for the runner, from head 9d853dc on this host (Docker 27.5.1).

The video is a 1x replay of a live asciinema recording. The four steps:

A probe runs in the same container as a repro job. The host env holds a fake ANTHROPIC_API_KEY and GH_TOKEN. The probe plants a symlink and a FIFO in its output.
explore without Docker is refused.
The mock-only repro phase runs by hand in the sandbox.
A SIGKILL of the launcher removes the running container.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $51.75

runner-dogfood.mp4

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d853dcbb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec0fa9cdda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d79c8e4973

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5fb22dddd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ThomasK33 · 2026-10-06T23:12:45Z

Parked at head c5fb22dddd. I have not changed the code or pushed since that head.

Why: the automatic normal review on this head (assessment 7) raised a new valid P2 finding in scope (thread PRRT_kwDOPxxmWM6psNLp). Assessment 8 (security review) found nothing. Under the agreed exception, a new valid finding means stop: no further push and no further assessment without a new decision.
Finding: dockerEnv() passes HOME and DOCKER_*, so the docker CLI reads the user's ~/.docker/config.json. The CLI copies its proxies settings into every docker run container and every build as HTTP_PROXY-style variables. That path bypasses containerEnv()'s allowlist and credential check, so an authenticated proxy URL would reach the container. On this host the client config has no proxies key, so nothing leaked here.
Proposed fix (not applied): run every docker command with DOCKER_CONFIG set to a fresh, empty private folder. Pass only DOCKER_HOST and DOCKER_CONTEXT, and drop HOME. Add a test that a proxies entry in the user's config never reaches the container env.
Assessments used: 8 of 9 (3 automatic rounds before the exception, then this pair). The final independent check (assessment 9) has not run.
Checks: Codex Comments fails on this head because of the open thread. Every other completed check passed when I read them.
Review history: 4, 3, 5 findings, then 1 after the scope reduction.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a213904c01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ThomasK33 · 2026-10-07T06:33:43Z

Parked at head a213904c01, final. Under the agreed rules there is no further push and no further assessment on this PR.

Why: the automatic normal review on this head (assessment 9) raised 4 new findings. Assessment 10 (security review) completed. Required and Codex Comments fail on this head. The final check (assessment 11) did not run.
Assessments used: 10 of 11.
Findings per round: 4, 3, 5, 1, then 4.

My triage of the 4 open threads. I did not change them or the code:

PRRT_kwDOPxxmWM6pyMjd (P1, no-agent rule before the host fallback): valid for this PR on its own. The rule that repros never take the agent fixture lives in reproRules.test.ts, which is in PR 3, not here. The runner can run by hand, so it must hold the rule itself.
PRRT_kwDOPxxmWM6pyMjf (P2, --output not checked on the host fallback): valid but small. The host fallback passes --output to e2e unchecked, as today's make targets do.
PRRT_kwDOPxxmWM6pyMji (P1, the checkout's Dockerfile builds with network and no build limits): partly valid. The checkout is the trusted side in this design: a hostile checkout can change the Makefile or launch.ts too. Still, the build is the one step that runs outside the job's limits, and the PR text does not say so.
PRRT_kwDOPxxmWM6pyMjl (P2, a SIGINT or SIGTERM during staging leaves the staged copy in /tmp): valid, small (a disk leak only).

Proposed next step (needs a new decision): fold 1 and 2 into PR 3, which already holds the repro rule and the make wiring, or redesign the runner to merge together with PR 3. 3 needs a stated threat-model decision. 4 is a small fix.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

- Read BUGBASH_AI=mock the way e2e.config.ts does. - A stop always closes the lifeline and kills the docker client, also when the removal fails. - A failed image build or same-host probe falls back to the host for exact-step runs in auto mode. - entry.ts waits until every other process is reaped before the export.

- Forward BUGBASH_SCENARIO to the container. - Staging and the export folder refuse a symlinked folder on the way (plainFolders). - The credential name check ignores case.

- Run only exact-step repro runs, in the sandbox and on the host fallback: `e2e run` with `--config e2e.config.ts` exactly once, selection and output options only (an allowlist), no positional files, no `--`, no `explore` argument, cwd tests/bugbash and a regular config file. Model-driven commands and configs wait for the provider proxy. - Do not forward BUGBASH_AI_REASON: it can hold a provider URL and response text. - The host fallback runs e2e in tests/bugbash, forwards SIGINT/SIGTERM and waits for teardown. - Exit codes keep the signal number (128 + n). - Container removal also matches this checkout's xum.bugbash.checkout label. - `docker info` without a daemon gives "no daemon answered", not "Docker Desktop".

…fig (#5714) The docker CLI copies the `proxies` entries of its client config into every build (as build args) and every container (as env). A proxy URL can hold a user and password, and that path skipped containerEnv()'s credential check. - One `docker context inspect` with the user's DOCKER_HOST, DOCKER_CONTEXT and DOCKER_CONFIG resolves the selected endpoint. Only a local unix socket passes. A remote, ssh or relative endpoint is refused, never swapped for the default socket. - Every other docker command (info, image inspect, build, the probe, the job, ps, rm) runs with PATH, DOCKER_HOST = that endpoint and DOCKER_CONFIG = a fresh, empty private folder that lives until the launcher exits. No HOME, no other DOCKER_* value, no credential or TLS file.

- No host fallback. Without usable Docker, with an unsupported endpoint, with a failed same-host probe or with the real app AI, the launcher refuses (exit 2). BUGBASH_SANDBOX is gone. The make targets keep their own host path, unchanged. - The image builds only from committed harness source: the Dockerfile at HEAD (a work-tree change refuses), sent on stdin with no build context, and one build arg (the installed Playwright version, checked to be a plain version). The build runs outside the job sandbox. - Cleanup of the job folder has an owner before staging starts: the launcher handles SIGINT and SIGTERM from then on, yields to the event loop after the synchronous steps, and exits 130 or 143. Only this job's folder is removed (it is created without `recursive`).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd37da52c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-07T07:12:01Z

+      ...bind(stageDir, "/repo"), ...bind(path.join(ROOT, "dist"), "/repo/dist"),
+      ...bind(path.join(ROOT, "node_modules"), "/repo/node_modules"),


Reject symlinked build-output mount sources

When either top-level dist or node_modules is a symlink, these mounts expose its host target to the untrusted repro, which can read and return those files through the export stream; the staged-source plainFolders() check does not cover these two sources. Docker's implementation resolves bind sources with filepath.EvalSymlinks, so validate both mount sources with lstat as plain directories or mount verified copies instead.

AGENTS.md reference: AGENTS.md:L152-L154

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-07T07:12:01Z

+export async function runInSandbox(run: SandboxRun): Promise<number> {
+  const dest = path.join(BUGBASH_DIR, run.exportDir);
+  if (fs.existsSync(dest)) throw new Refusal(`${run.exportDir} exists: remove it first`);
+  const image = ensureImage();


Install termination handling before building the image

When the image is missing and a supervisor sends SIGINT or SIGTERM during the potentially 20-minute docker build, ensureImage() runs before the handlers at line 307 are installed, so default signal termination kills the launcher without forwarding the signal or waiting for the Docker child; that build can continue orphaned and its temporary Docker client directory is not cleaned. Fresh evidence beyond the earlier preflight-cleanup thread is that image construction still precedes all cleanup ownership; register termination handling before this call and dispose of the build process as well.

AGENTS.md reference: AGENTS.md:L159-L165

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-07T07:12:01Z

+  if (why != null) throw new Refusal(`${why}. The launcher runs jobs in the sandbox only.`);
+  const command = ["node", "../../node_modules/e2e/dist/cli/bin.js", ...e2eArgs];
+  // The app log goes into the output folder, so that it comes back with the report.
+  const env = { BUGBASH_APP_LOG: process.env.BUGBASH_APP_LOG ?? `${output}/app.log` };


Keep inherited app logs inside the exported directory

When BUGBASH_APP_LOG is already set, this preserves that path even though the export stream returns only output; for example, run.ts deliberately uses the sibling path ${outRel}.app.log, which remains in the container's .e2e tmpfs and disappears when the container exits while the evidence is still reported as complete. Override the variable with ${output}/app.log, or validate and rebase inherited paths beneath output, so failures retain the app log needed for triage.

Useful? React with 👍 / 👎.

ThomasK33 · 2026-10-07T07:12:52Z

Parked for re-planning at head fd37da52c0. Under the user's waiver there is no further repair push on this PR.

Why: the automatic normal review on this head (assessment 11) raised 3 new P2 findings. Assessment 12 (security review) completed. Required and Codex Comments fail on this head. The final check (assessment 13) did not run.
Assessments used: 12 of 13.
Findings per round: 4, 3, 5, 1, 4, then 3. The earlier 4 threads are fixed and resolved on this head.

My triage of the 3 open threads. I did not change them or the code:

PRRT_kwDOPxxmWM6py4cr (symlinked dist or node_modules as a mount source): valid. Docker resolves a bind source through symlinks, and plainFolders() covers only the staged copy and the export folder. A fix is an lstat check on both mount sources.
PRRT_kwDOPxxmWM6py4cv (a signal during docker build): valid. The handlers are installed after ensureImage(). I chose this on purpose so that a signal does not wait for a 20-minute synchronous build, but the build is then orphaned and the empty private client folder stays. A fix needs an asynchronous build that the launcher can stop.
PRRT_kwDOPxxmWM6py4cy (an inherited BUGBASH_APP_LOG outside the output folder): valid, low. Only run.ts sets it (for explore, which this runner refuses), but a value set by hand is not checked. A fix is to always set it to <output>/app.log.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

## Summary Model-driven bug-bash runs now refuse to start on the host. `make bug-bash`, `make mcp-apps-e2e`, `tests/bugbash/run.ts`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts` exit 2 with a message that names coder#5714. They refuse before any app, browser, child process or model starts. Exact-step repros (`make test-bugbash-repros`, `make test-bugbash-known-failures`) run exactly as before. ## Background In coder#5714 the user chose to **pause** model-driven runs on the host until they run in the bug-bash sandbox. In such a run a model reads untrusted page and AI text and picks the UI actions, and the app it drives can run commands on this host. The sandbox runner (coder#5802) is parked for a re-plan, so this PR delivers the pause on its own. It is based on `main`, is not stacked on coder#5802, and imports no runner code. ## Implementation 1. `tests/bugbash/hostPause.ts` holds the rule. It has no override: no env var or flag turns it off. 2. `e2e.config.ts` checks `process.argv` as its first statement. Every e2e command loads the config in its own process before it starts an app, a browser or a model. Only the e2e CLI with `run` or `list`, and the e2e worker that such a run forks, may load it. Anything else fails closed: `explore`, `mcp`, other commands, and a config loaded by any other script. 3. `e2e.mcpapps.config.ts` refuses every command, `run` included, because its suite drives each flow with `agent.act`. Its first import is `mcpapps/hostPause.ts`, so the refusal comes before `e2e.config.ts` runs (that config can throw first, for example on an unresolved app AI mode). 4. `run.ts` refuses at the top of `main()`, before the app-model probe and before it spawns e2e. 5. The persona in `e2e.config.ts` holds no explorer model during the pause. An `agent.*` step in a repro therefore fails with `MODEL_UNAVAILABLE` and makes no model request. e2e has no default model and no env fallback. `reproRules.test.ts` also fails on a repro, in any subfolder, that takes the `agent` fixture. 6. The `bug-bash` and `mcp-apps-e2e` make targets no longer build first, so they refuse at once. 7. The docs/AGENTS.md bug-bash sections describe the pause (builtInSkillContent regenerated). The host restrictions `XUM_DISABLE_AGENT_TOOLS`, `XUM_DISABLE_TERMINALS` and `XUM_DISABLE_PROJECT_AUTOMATION` are unchanged. Limit: for an `agent.*` step inside an exact-step repro, e2e starts the app before the test reaches the step. The step then fails with `MODEL_UNAVAILABLE` and makes 0 model requests. No model ever picks an action there. ## Validation 1. `hostPause.test.ts` runs each refused path as an async child with a fake app command, a fake e2e node and a counting fake provider. Each path exits 2, starts nothing and makes 0 model requests: e2e explore, e2e explore with MCP Apps, e2e run with MCP Apps, e2e list with MCP Apps and no app AI mode, e2e explore with real app AI, and run.ts. A control `e2e list` still lists the repros. 2. Mutation check: 10 mutants, all killed (refusal returns null, `explore` allowed, unknown loader allowed, worker check widened, run.ts guard removed, run.ts probes before its guard, MCP Apps guard removed, MCP Apps pause imported after the base config, e2e.config guard removed, non-recursive repro scan). 3. `make test-bugbash-repros`: 42 selected, 34 + 8 passed. `make test-bugbash-known-failures`: both known failures fail on their own `ASSERTION_FAILED` assertions, as on `main`. 4. Probe: a scratch repro with `agent.act` failed with `MODEL_UNAVAILABLE`, 0 requests to a counting provider. With the persona model restored (mutant), the same probe sent 6 requests. Dogfood: each entry point refuses, and the exact-step listing still works. ![Host pause dogfood](https://github.com/user-attachments/assets/37106669-91df-41f0-b6b5-c48aa083f440) https://github.com/user-attachments/assets/243823ee-7e68-4215-a2cd-60f5269e8702 ## Risks Low. The change touches only bug-bash test tooling, make targets and docs. No product code changes. If the argv check were wrong for exact-step runs, the repro targets would fail loudly. They pass. Refs coder#5714 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$85.00`_

ThomasK33 · 2026-10-08T13:46:52Z

Closing as superseded by the B1 delivery of the container runner. Its tested parts moved into these merged PRs:

🤖 tests: sandbox runner owns its job and stops whole process groups #5878 7ba0f93
🤖 tests: inputs library for the bug-bash sandbox (staging, mount sources, env, app log) #5880 cd2837a
🤖 tests: launch command for the bug-bash sandbox (exact-step repros in a container) #5881 27c8b92

The runner now pulls a published image by digest (#5818, #5875) instead of building on the host. Remaining hardening items are tracked for B1 PR 4 (#5873, #5877 and the #5881 body).

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…rom main (coder#5818) ## Summary This is PR 1 of 4 for the approved B1 plan (coder#5714). It adds a bug-bash sandbox image and the one script that builds it. Only a manual run of a CI workflow on `main` publishes the image. No runner uses the image yet. PR 2 adds the runner library and `image.json` with the first digest. ## Background The parked runner (coder#5802) built its image on each host, synchronously, before its signal handlers existed. The B1 plan replaces that build with an image that CI publishes. Runners pull it by the digest in a reviewed file. This PR covers only the image side. ## Implementation 1. `tests/bugbash/sandbox/Dockerfile` installs tools only: node 22, bun 1.3.12, git, and Chromium for Playwright. Both base images are pinned by digest. The labels `org.xum.bugbash.inputs` and `org.xum.bugbash.playwright-core` name its inputs. 2. `tests/bugbash/sandbox/build.sh` has three modes, and the Make targets call it: - `make bugbash-sandbox-key` (`--key`) prints the inputs key. The key is the sha256 of three values: the Dockerfile hash, the build.sh hash, and the playwright-core version of `@e2e-dev/web` in `bun.lock` (1.63.0, Chromium 1243). The root 1.57.0 copy does not count. The script refuses when an input is not committed or differs from `HEAD`. - `make bugbash-sandbox-image` makes a local linux/amd64 build and prints the image ID. - `make bugbash-sandbox-publish` (`--push`) is for the workflow only. It always builds fresh: it never checks the registry and never reuses an image. It pushes a new tag, `inputs-<key16>-<commit12>-<UTC time>`, so a publish overwrites no tag. It prints the `image.json` record, with the image name and digest taken from the build's own metadata. - Every build uses an empty context, `--no-cache --pull`, no provenance or SBOM manifests, and no build args from the environment. The scripts work with the bash 3.2 that macOS ships, and they fall back to `shasum` when `sha256sum` is missing. 3. `.github/workflows/bugbash-sandbox-image.yml`: - The default is `permissions: {}`. - **Pull requests** that touch the image inputs, the `Makefile` or the workflow run `Build (no push)` with `contents: read` only, and report the size. - **Publish** runs only on a `workflow_dispatch` run on `main` (`github.ref == 'refs/heads/main'`). It checks out that run's exact `github.sha` without persisted credentials. It is the only job with `packages: write`, `id-token: write` and `attestations: write`. It pushes, writes `image.json` to the job summary, and attests exactly the name and digest from that `image.json`. No cache and no artifact from another run is an input. No `push` trigger exists. Nothing deletes an image. No file under `src/` changes. The coder#5815 host pause stays. ## Publishing is manual (changes to the B1 plan) 1. No merge publishes anything. A person runs the workflow on `main`, then opens a pull request that copies the digest from the job summary into `image.json`. No workflow writes to the repo. 2. After a change to `bun.lock` (the `@e2e-dev/web` playwright-core version), the Dockerfile or `build.sh` merges, local runs refuse with the stale inputs-key error. They refuse until someone publishes a new image and its digest pull request merges. 3. One inputs key no longer maps to one digest: each publish builds fresh, so two publishes of the same key give two digests. The reviewed digest in `image.json` stays the trust anchor. 4. Nothing planned for PR 2 or PR 3 depends on one key mapping to one digest. PR 2 pulls the `image.json` digest and refuses when the image label differs from the checkout's key. PR 4's CI picks either the `image.json` digest or a local candidate image. None of them looks up an image by its key or its tag. ### Package visibility (held for approval) 1. GitHub docs say that a container package is private by default when it is first published. Its visibility changes on the package settings page. 2. A public package cannot be made private again. 3. Two third-party reports say the REST API cannot set package visibility. Neither I nor the coordinator verified this. 4. PR 2's gate is an anonymous pull, so the package must be public before PR 2 can pass it. 5. Nothing deletes old images, so every publish adds an image to the package. The image is 1.43 GB in total (1,425,909,942 bytes in the local Docker). That is the whole image size, not new registry storage per publish: the registry shares layers between images, and I did not measure the storage. ## Validation 1. `build.test.ts` runs the real `build.sh` in throwaway git repos: - The key changes with each of its 3 inputs and stays the same otherwise. - An uncommitted edit refuses, and a lock file without the `@e2e-dev/web` copy refuses. - With a fake `docker`, two `--push` runs of one key make two push builds and no registry call. They report the name and digest from the build metadata. A mutant that adds a registry lookup fails this test. 2. Local build with `make bugbash-sandbox-image` on this host: 45 s wall time, 1,425,909,942 bytes, linux/amd64. The label `org.xum.bugbash.inputs` equals the output of `make bugbash-sandbox-key`. A `--network none` run as uid 1000 found bun 1.3.12, node v22.23.3, git 2.39.5, `chromium-1243` and `chromium_headless_shell-1243`. 3. CI `Build (no push)` on the previous head built 1,425,793,449 bytes. 4. `make static-check`, `make lint-actions` (actionlint and zizmor) and `bun test ./tests/bugbash` pass (43 tests). Not measured yet: the GHCR pull time. It needs the first publish, which waits for approval. ## Risks The risk to the product is low. This PR changes only test tooling, three Make targets and a new workflow. The publish job is the only place with registry write access, and it runs only on a manual run on `main`. Refs coder#5714 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$104.23`_

…e digest (coder#5875) ## Summary This is PR 2 of 4 for the approved B1 plan (coder#5714). It adds the runner library of the bug-bash sandbox, plus `image.json` with the first published digest. The library has no entry point, so nothing can run a job yet. PR 3 adds staging, the mount checks, the app log and the launch command. ## Background coder#5818 added the image workflow, and a manual publish ran on `main`: [run 37759827317](https://github.com/coder/xum/actions/runs/37759827317) (`workflow_dispatch`, `main`, `752a624e15`). This PR records that digest in a reviewed file. That is the "a person opens the digest PR" step of the plan. It also adds the code that uses the digest. ## Implementation 1. `tests/bugbash/sandbox/image.json` pins the published image. I copied every value from tool output, not by hand: - The digest comes from the publish run's Attest step. It equals the sha256 of the registry manifest. - The key and the playwright-core version come from the image's labels. - `publishedFrom` is the run's head SHA. - The key equals `make bugbash-sandbox-key` on this branch. 2. `tests/bugbash/sandbox/runner.ts` has two parts. `readImageLock()` validates `image.json`. `Session` does the work: - **ensureImage()** first compares the checkout's inputs key (`build.sh --key`) with `image.json`. A mismatch refuses as a stale image, before any docker command. Next it finds the local daemon. If the image is missing, it pulls only `name@digest`. Then it checks that the image label `org.xum.bugbash.inputs` equals the key. An empty `docker images` list with exit 0 means "absent". Any query failure refuses, so a failure is never read as "absent". The runner never builds. - **Docker preflight** comes from coder#5802. It refuses non-Linux hosts, root, remote or ssh endpoints, Docker Desktop and rootless Docker. After `docker context inspect`, every docker command gets only `PATH`, `DOCKER_HOST` and an empty private `DOCKER_CONFIG`. - **Children:** every command is an async child that the session tracks. The entry point (PR 3) will abort the session's `AbortSignal` from its SIGINT and SIGTERM handlers. Then running children get SIGTERM, and SIGKILL after 5 s. Job commands started after that refuse with `Stopped`. - **cleanup(job)** is one idempotent promise for success, error and stop. It waits for every child. It removes only the container that matches the name and both labels (owner and checkout), and it confirms the removal. It reports `unknown: …`, never success, when it cannot read the container state. It removes the private client folder. It never removes an image. ## Validation 1. Gate, the anonymous pull: in a fresh `docker:27-dind` with no client config and no images, `docker pull ghcr.io/coder/xum-bugbash-sandbox@sha256:61adf1a543f2b1cc1373506556eefad52176aabd5bb1e491927eef82bde8fa73` exited 0 in 30,228 ms and gave 1,425,818,783 bytes. An anonymous `imagetools inspect` (empty `DOCKER_CONFIG`) gave the same digest, so the package is public. 2. Gate, the stale-key refusal: `runner.test.ts` "a stale inputs key refuses before any docker command". 3. `runner.test.ts` runs the real `build.sh` in throwaway git repos with a fake `docker`. It covers: - a pull by digest when the image is missing, and no pull when it is present; - a refusal on a label mismatch, on an image query failure, on a remote endpoint, with no daemon, on Docker Desktop and on rootless Docker; - the private client env; - a stop during a pull that ignores SIGTERM: it ends with `Stopped` after SIGKILL, new commands refuse, and cleanup still removes the job container with the full filter; - cleanup that reports `unknown` when `ps` fails, and that returns the same promise on a second call. 4. Mutation check: 10 mutants, all fail the tests. They are: no stale-key check, query failure read as absent, no label check, no SIGKILL escalation, cleanup gated by the stop, unknown reported as removed, user env passed to docker, cleanup not idempotent, remote endpoint allowed, and the checkout label filter dropped. 5. `make static-check` passes. `bun test ./tests/bugbash` passes (55 tests). Size: 3 files, +454 lines with tests. ## Risks Low. This PR adds test tooling only, and nothing calls the library yet. No file under `src/` changes. The coder#5815 host pause stays. Refs coder#5714 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$111.52`_

ThomasK33 added this pull request to stack #5803 October 6, 2026 22:05

chatgpt-codex-connector Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/entry.ts

Base automatically changed from tests/5714-sandbox-export to main October 6, 2026 22:24

ThomasK33 force-pushed the tests/5714-sandbox-runner branch from 9d853dc to f537545 Compare October 6, 2026 22:24

ThomasK33 mentioned this pull request Oct 6, 2026

tests: bug-bash sandbox export stream follow-ups (#5800 final check) #5804

Open

chatgpt-codex-connector Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/bugbash/sandbox/launch.ts

Comment thread tests/bugbash/sandbox/launch.ts

Comment thread tests/bugbash/sandbox/launch.ts

chatgpt-codex-connector Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/launch.ts

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/entry.ts Outdated

chatgpt-codex-connector Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread tests/bugbash/sandbox/launch.ts Outdated

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread tests/bugbash/sandbox/launch.ts

Comment thread tests/bugbash/sandbox/launch.ts Outdated

Comment thread tests/bugbash/sandbox/launch.ts

Comment thread tests/bugbash/sandbox/launch.ts

ThomasK33 added 6 commits October 7, 2026 07:00

tests: container runner for the bug-bash sandbox (#5714)

02bd1c1

tests: sandbox runner review fixes, round 2 (#5714)

a167cc0

- Forward BUGBASH_SCENARIO to the container. - Staging and the export folder refuse a symlinked folder on the way (plainFolders). - The credential name check ignores case.

ThomasK33 force-pushed the tests/5714-sandbox-runner branch from a213904 to fd37da5 Compare October 7, 2026 07:01

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

ThomasK33 marked this pull request as draft October 7, 2026 07:25

This was referenced Oct 7, 2026

🤖 tests: pause model-driven bug-bash runs on the host #5815

Merged

🤖 tests: run the bug-bash app and browser in a container sandbox #5714

Open

🤖 tests: a digest-pinned bug-bash sandbox image, published manually from main #5818

Merged

ThomasK33 mentioned this pull request Oct 8, 2026

🤖 tests: runner library for the bug-bash sandbox, with the first image digest #5875

Merged

ThomasK33 closed this Oct 8, 2026

		...bind(stageDir, "/repo"), ...bind(path.join(ROOT, "dist"), "/repo/dist"),
		...bind(path.join(ROOT, "node_modules"), "/repo/node_modules"),

Conversation

ThomasK33 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Implementation

Validation

Risks

Uh oh!

chatgpt-codex-connector Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

ThomasK33 commented Oct 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ThomasK33 commented Oct 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ThomasK33 commented Oct 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

ThomasK33 commented Oct 7, 2026

Uh oh!

ThomasK33 commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

ThomasK33 commented Oct 6, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 6, 2026 •

edited

Loading