Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ Freely make breaking changes, and reorganize / cleanup IPC as needed.
- Host pause (#5714): model-driven runs (`make bug-bash`, `make mcp-apps-e2e`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts`) run only inside the bug-bash sandbox. On the host, `make bug-bash` and `make mcp-apps-e2e` are the launchers (`tests/bugbash/run.ts`, `tests/bugbash/sandbox/launch.ts`): they start the sandbox containers. A model-driven e2e process on the host still refuses with exit 2 before any app, browser or model starts (`tests/bugbash/hostPause.ts`). The launchers need `BUGBASH_BUDGET_USD` plus `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` on the host, and the app of every model-driven job runs with agent tools, terminals and project automation off. `make bug-bash` resolves the app AI once on the host (rules below), with its probe through the run's budget, and a real app AI reaches its model only through each job's proxy. Outside a sandboxed `e2e explore`, the agents of `e2e.config.ts` hold no model, so an `agent.*` step in a repro fails with `MODEL_UNAVAILABLE` and makes no model request. No override exists. Exact-step repros run as before.
- `BUGBASH_BUDGET_USD` caps one launch for `make mcp-apps-e2e`, and the whole run (every charter's job and the app-AI probe together) for `make bug-bash`. It is a list-price cap from the reported usage, not a billing cap: the key's own limit is the backstop.
- Sandbox exit codes (`tests/bugbash/sandbox/launch.ts`): 2 refused, 3 a container's state is unknown after cleanup, 4 incomplete evidence, 5 a proxy fault, 6 the job's own e2e exited 3, 130 or 143 stopped. Exit 3 outranks a stop, and a stop outranks 5. Each job writes its proxy record to `<output>.proxy.jsonl` on the host.
- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches one self-check job (`tests/bugbash/sandbox/selfCheck.ts`) against a fake upstream, with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:<model>`, and with `BUGBASH_SCENARIO` set.
- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches three sandbox jobs against a fake upstream: the self-check (`tests/bugbash/sandbox/selfCheck.ts`) and two export fixtures (a path traversal and an oversized frame), with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:<model>`, and with `BUGBASH_SCENARIO` set.
- Leftovers: every launch lists the job containers of its checkout. `make bug-bash-sandbox-recover` removes only those whose launcher is provably dead on this boot. It leaves every container whose owner it cannot tell (another boot, another PID namespace, `/proc` with `hidepid`, a failed inspect) and then exits 3. Check such a container's `xum.bugbash.owner` and `xum.bugbash.checkout` labels with `docker inspect`, then `docker rm -f <id>` it by hand. A SIGKILLed launcher also leaves its job folder (`$TMPDIR/xum-bugbash-sandbox/<checkout>/<job>`, with the proxy's private socket folder inside) and a `xum-bugbash-docker-*` client folder in `$TMPDIR`: remove both by hand.
- `make bug-bash` runs AI explorers (`e2e explore` charters) against seeded `xum server`s, each in its own sandbox container (see above). This section is the repo's rule set: `.xum/skills/bug-bash/` is vendored, so do not edit it; where the two differ (repros are tracked), follow this section.
- A finding is a bug only once a repro fails for the reported reason. Put it in `tests/bugbash/repros/<slug>.e2e.ts` (tracked) with exact `screen`/`browser`/`expect` steps, never `agent.*`, so it needs no model key and gives the same result every run. `tests/bugbash/reproRules.test.ts` fails on a repro that takes the `agent` fixture.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2303,7 +2303,7 @@ export const BUILTIN_SKILL_FILES: Record<string, Record<string, string>> = {
"- Host pause (#5714): model-driven runs (`make bug-bash`, `make mcp-apps-e2e`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts`) run only inside the bug-bash sandbox. On the host, `make bug-bash` and `make mcp-apps-e2e` are the launchers (`tests/bugbash/run.ts`, `tests/bugbash/sandbox/launch.ts`): they start the sandbox containers. A model-driven e2e process on the host still refuses with exit 2 before any app, browser or model starts (`tests/bugbash/hostPause.ts`). The launchers need `BUGBASH_BUDGET_USD` plus `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` on the host, and the app of every model-driven job runs with agent tools, terminals and project automation off. `make bug-bash` resolves the app AI once on the host (rules below), with its probe through the run's budget, and a real app AI reaches its model only through each job's proxy. Outside a sandboxed `e2e explore`, the agents of `e2e.config.ts` hold no model, so an `agent.*` step in a repro fails with `MODEL_UNAVAILABLE` and makes no model request. No override exists. Exact-step repros run as before.",
"- `BUGBASH_BUDGET_USD` caps one launch for `make mcp-apps-e2e`, and the whole run (every charter's job and the app-AI probe together) for `make bug-bash`. It is a list-price cap from the reported usage, not a billing cap: the key's own limit is the backstop.",
"- Sandbox exit codes (`tests/bugbash/sandbox/launch.ts`): 2 refused, 3 a container's state is unknown after cleanup, 4 incomplete evidence, 5 a proxy fault, 6 the job's own e2e exited 3, 130 or 143 stopped. Exit 3 outranks a stop, and a stop outranks 5. Each job writes its proxy record to `<output>.proxy.jsonl` on the host.",
"- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches one self-check job (`tests/bugbash/sandbox/selfCheck.ts`) against a fake upstream, with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:<model>`, and with `BUGBASH_SCENARIO` set.",
"- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches three sandbox jobs against a fake upstream: the self-check (`tests/bugbash/sandbox/selfCheck.ts`) and two export fixtures (a path traversal and an oversized frame), with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:<model>`, and with `BUGBASH_SCENARIO` set.",
"- Leftovers: every launch lists the job containers of its checkout. `make bug-bash-sandbox-recover` removes only those whose launcher is provably dead on this boot. It leaves every container whose owner it cannot tell (another boot, another PID namespace, `/proc` with `hidepid`, a failed inspect) and then exits 3. Check such a container's `xum.bugbash.owner` and `xum.bugbash.checkout` labels with `docker inspect`, then `docker rm -f <id>` it by hand. A SIGKILLed launcher also leaves its job folder (`$TMPDIR/xum-bugbash-sandbox/<checkout>/<job>`, with the proxy's private socket folder inside) and a `xum-bugbash-docker-*` client folder in `$TMPDIR`: remove both by hand.",
"- `make bug-bash` runs AI explorers (`e2e explore` charters) against seeded `xum server`s, each in its own sandbox container (see above). This section is the repo's rule set: `.xum/skills/bug-bash/` is vendored, so do not edit it; where the two differ (repros are tracked), follow this section.",
"- A finding is a bug only once a repro fails for the reported reason. Put it in `tests/bugbash/repros/<slug>.e2e.ts` (tracked) with exact `screen`/`browser`/`expect` steps, never `agent.*`, so it needs no model key and gives the same result every run. `tests/bugbash/reproRules.test.ts` fails on a repro that takes the `agent` fixture.",
Expand Down
Loading