diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 56155b81a7c..04cb6d59eae 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -250,7 +250,7 @@ Freely make breaking changes, and reorganize / cleanup IPC as needed. - Host pause (#5714): model-driven runs (`make bug-bash`, `make mcp-apps-e2e`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts`) run only inside the bug-bash sandbox. On the host, `make bug-bash` and `make mcp-apps-e2e` are the launchers (`tests/bugbash/run.ts`, `tests/bugbash/sandbox/launch.ts`): they start the sandbox containers. A model-driven e2e process on the host still refuses with exit 2 before any app, browser or model starts (`tests/bugbash/hostPause.ts`). The launchers need `BUGBASH_BUDGET_USD` plus `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` on the host, and the app of every model-driven job runs with agent tools, terminals and project automation off. `make bug-bash` resolves the app AI once on the host (rules below), with its probe through the run's budget, and a real app AI reaches its model only through each job's proxy. Outside a sandboxed `e2e explore`, the agents of `e2e.config.ts` hold no model, so an `agent.*` step in a repro fails with `MODEL_UNAVAILABLE` and makes no model request. No override exists. Exact-step repros run as before. - `BUGBASH_BUDGET_USD` caps one launch for `make mcp-apps-e2e`, and the whole run (every charter's job and the app-AI probe together) for `make bug-bash`. It is a list-price cap from the reported usage, not a billing cap: the key's own limit is the backstop. - Sandbox exit codes (`tests/bugbash/sandbox/launch.ts`): 2 refused, 3 a container's state is unknown after cleanup, 4 incomplete evidence, 5 a proxy fault, 6 the job's own e2e exited 3, 130 or 143 stopped. Exit 3 outranks a stop, and a stop outranks 5. Each job writes its proxy record to `.proxy.jsonl` on the host. -- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches one self-check job (`tests/bugbash/sandbox/selfCheck.ts`) against a fake upstream, with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:`, and with `BUGBASH_SCENARIO` set. +- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches three sandbox jobs against a fake upstream: the self-check (`tests/bugbash/sandbox/selfCheck.ts`) and two export fixtures (a path traversal and an oversized frame), with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:`, and with `BUGBASH_SCENARIO` set. - Leftovers: every launch lists the job containers of its checkout. `make bug-bash-sandbox-recover` removes only those whose launcher is provably dead on this boot. It leaves every container whose owner it cannot tell (another boot, another PID namespace, `/proc` with `hidepid`, a failed inspect) and then exits 3. Check such a container's `xum.bugbash.owner` and `xum.bugbash.checkout` labels with `docker inspect`, then `docker rm -f ` it by hand. A SIGKILLed launcher also leaves its job folder (`$TMPDIR/xum-bugbash-sandbox//`, with the proxy's private socket folder inside) and a `xum-bugbash-docker-*` client folder in `$TMPDIR`: remove both by hand. - `make bug-bash` runs AI explorers (`e2e explore` charters) against seeded `xum server`s, each in its own sandbox container (see above). This section is the repo's rule set: `.xum/skills/bug-bash/` is vendored, so do not edit it; where the two differ (repros are tracked), follow this section. - A finding is a bug only once a repro fails for the reported reason. Put it in `tests/bugbash/repros/.e2e.ts` (tracked) with exact `screen`/`browser`/`expect` steps, never `agent.*`, so it needs no model key and gives the same result every run. `tests/bugbash/reproRules.test.ts` fails on a repro that takes the `agent` fixture. diff --git a/src/node/services/agentSkills/builtInSkillContent.generated.ts b/src/node/services/agentSkills/builtInSkillContent.generated.ts index 0d18f6c27ae..fbb183911dd 100644 --- a/src/node/services/agentSkills/builtInSkillContent.generated.ts +++ b/src/node/services/agentSkills/builtInSkillContent.generated.ts @@ -2303,7 +2303,7 @@ export const BUILTIN_SKILL_FILES: Record> = { "- Host pause (#5714): model-driven runs (`make bug-bash`, `make mcp-apps-e2e`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts`) run only inside the bug-bash sandbox. On the host, `make bug-bash` and `make mcp-apps-e2e` are the launchers (`tests/bugbash/run.ts`, `tests/bugbash/sandbox/launch.ts`): they start the sandbox containers. A model-driven e2e process on the host still refuses with exit 2 before any app, browser or model starts (`tests/bugbash/hostPause.ts`). The launchers need `BUGBASH_BUDGET_USD` plus `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` on the host, and the app of every model-driven job runs with agent tools, terminals and project automation off. `make bug-bash` resolves the app AI once on the host (rules below), with its probe through the run's budget, and a real app AI reaches its model only through each job's proxy. Outside a sandboxed `e2e explore`, the agents of `e2e.config.ts` hold no model, so an `agent.*` step in a repro fails with `MODEL_UNAVAILABLE` and makes no model request. No override exists. Exact-step repros run as before.", "- `BUGBASH_BUDGET_USD` caps one launch for `make mcp-apps-e2e`, and the whole run (every charter's job and the app-AI probe together) for `make bug-bash`. It is a list-price cap from the reported usage, not a billing cap: the key's own limit is the backstop.", "- Sandbox exit codes (`tests/bugbash/sandbox/launch.ts`): 2 refused, 3 a container's state is unknown after cleanup, 4 incomplete evidence, 5 a proxy fault, 6 the job's own e2e exited 3, 130 or 143 stopped. Exit 3 outranks a stop, and a stop outranks 5. Each job writes its proxy record to `.proxy.jsonl` on the host.", - "- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches one self-check job (`tests/bugbash/sandbox/selfCheck.ts`) against a fake upstream, with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:`, and with `BUGBASH_SCENARIO` set.", + "- Sandbox checks (#5931): `make bug-bash-sandbox-check` launches three sandbox jobs against a fake upstream: the self-check (`tests/bugbash/sandbox/selfCheck.ts`) and two export fixtures (a path traversal and an oversized frame), with no key and $0 spend. It fails unless every isolation, proxy and export check passes. `make test-bugbash-repros-sandbox` runs the repros of fixed bugs through the launcher, on the mock app AI. CI's `Test / Bug-bash sandbox` job runs both without secrets, and `Required` needs it. The launcher refuses (exit 2) outside Linux, as root, with Docker Desktop, rootless Docker or a Docker endpoint that is not a local socket. A model-driven job also refuses without `BUGBASH_BUDGET_USD` or the Anthropic key and base URL, with a `BUGBASH_MODEL` that is not `anthropic:`, and with `BUGBASH_SCENARIO` set.", "- Leftovers: every launch lists the job containers of its checkout. `make bug-bash-sandbox-recover` removes only those whose launcher is provably dead on this boot. It leaves every container whose owner it cannot tell (another boot, another PID namespace, `/proc` with `hidepid`, a failed inspect) and then exits 3. Check such a container's `xum.bugbash.owner` and `xum.bugbash.checkout` labels with `docker inspect`, then `docker rm -f ` it by hand. A SIGKILLed launcher also leaves its job folder (`$TMPDIR/xum-bugbash-sandbox//`, with the proxy's private socket folder inside) and a `xum-bugbash-docker-*` client folder in `$TMPDIR`: remove both by hand.", "- `make bug-bash` runs AI explorers (`e2e explore` charters) against seeded `xum server`s, each in its own sandbox container (see above). This section is the repo's rule set: `.xum/skills/bug-bash/` is vendored, so do not edit it; where the two differ (repros are tracked), follow this section.", "- A finding is a bug only once a repro fails for the reported reason. Put it in `tests/bugbash/repros/.e2e.ts` (tracked) with exact `screen`/`browser`/`expect` steps, never `agent.*`, so it needs no model key and gives the same result every run. `tests/bugbash/reproRules.test.ts` fails on a repro that takes the `agent` fixture.",