Skip to content

Commit 67da7c8

Browse files
authored
🤖 tests: list bug-bash sandbox crash leftovers and add a recover target (#6072)
## Summary Lists the job containers that crashed launches left behind, and adds `make bug-bash-sandbox-recover` to remove the ones whose launcher is gone. This is PR C2 of the plan to run the model-driven bug bash in the sandbox, stacked on #6069 (C). It must land before B1, which activates model-driven jobs. Refs #5714 Fixes #5882 ## What changes 1. Every launch lists the job containers of its checkout, found by the `xum.bugbash.checkout` label, right after `docker info`. Each line shows the owner state: `dead`, `alive` or `cannot tell`. The launch removes nothing. 2. `make bug-bash-sandbox-recover` (`launch.ts --recover`) prints the Docker endpoint first. It removes a container only when both hold: - Its name matches this checkout's job pattern `^xbb-<checkout6>-[0-9a-f]{6}$`. - Its owner is dead: the same boot ID and PID namespace, and a PID that is gone or now has another start time (a reused PID). 3. Each removal is by ID, after the same name and label check that cleanup uses (#6069). Every other container is listed and left. A removal it cannot prove exits 3. 4. The owner label names the launcher, not the Docker daemon, so removal happens only when a person runs the target on purpose. That is why the target prints the endpoint before it removes anything. ## Real-Docker dogfood (Docker 27.5.1) I created four test containers with names that only this check uses, then ran a launch (contract case (a) of #6069) and the recover target twice. | Container | Owner | Launch listing | First recover | Second recover (after the owner ended) | | --- | --- | --- | --- | --- | | `xbb-0b6a51-c2dead` | dead | `owner dead` | removed by ID | (already gone) | | `xbb-0b6a51-c2a11e` | a live `sleep` | `owner alive` | left | removed | | `xbb-0b6a51-c2live` | the same live `sleep` | `owner alive` | left: not a job name | left: owner dead, not a job name | | `xbbprobe-c2-other` | dead, other checkout | not listed | not listed | not listed | The launch still ended `cleanup: removed` for its own job container, and both recover runs exited 0. I removed the two remaining test containers by their exact IDs afterwards. <details> <summary>1-launch.log</summary> ```text case a: launcher start 2026-10-10T15:39:09.174Z checkout=0b6a51dd309e +1.74s sandbox leftover: xbb-0b6a51-c2a11e owner alive (make bug-bash-sandbox-recover) +1.74s sandbox leftover: xbb-0b6a51-c2live owner alive (make bug-bash-sandbox-recover) +1.75s sandbox leftover: xbb-0b6a51-c2dead owner dead (make bug-bash-sandbox-recover) +2.11s sandbox xbb-0b6a51-28cbc7 staged 3414 files +2.11s sandbox xbb-0b6a51-28cbc7 starts: --network none, mock app AI +2.11s ACTION: SIGTERM to the launcher right after the spawn of docker run +2.71s sandbox entry: the launcher is gone, so the job stops +2.79s sandbox xbb-0b6a51-28cbc7 exit 137, 0 files, incomplete: no end frame +2.79s sandbox xbb-0b6a51-28cbc7 cleanup: container fcd31d124888, ID from the cidfile +3.03s sandbox xbb-0b6a51-28cbc7 cleanup: removed +3.03s sandbox stopped: SIGTERM +3.03s launcher exit 143 containers of this checkout after exit: 2dc93aaaf0e1 xbb-0b6a51-c2a11e Created bcefb43a91bc xbb-0b6a51-c2live Created d1edf4985d9f xbb-0b6a51-c2dead Created export files: 0 ``` </details> <details> <summary>2-recover.log</summary> ```text $ make bug-bash-sandbox-recover sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e sandbox recover: left xbb-0b6a51-c2a11e: owner alive sandbox recover: left xbb-0b6a51-c2live: owner alive, not a job name sandbox xbb-0b6a51-c2dead cleanup: container d1edf4985d9f, ID from the leftover list sandbox recover: xbb-0b6a51-c2dead removed exit 0 ``` </details> <details> <summary>4-recover-again.log</summary> ```text $ make bug-bash-sandbox-recover (after the live owner ended) sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e sandbox xbb-0b6a51-c2a11e cleanup: container 2dc93aaaf0e1, ID from the leftover list sandbox recover: xbb-0b6a51-c2a11e removed sandbox recover: left xbb-0b6a51-c2live: owner dead, not a job name exit 0 ``` </details> ## Validation 1. The tests ran failing-first: the test file failed to load, because `launch.ts` had no `ownerState` or `recover` export. 2. New tests: `ownerState` (dead, alive, a reused PID, another boot, another PID namespace, malformed labels); a launch lists every leftover and removes nothing; recover removes only dead owners with a job name, by ID, and pulls or runs nothing; recover exits 3 when a removal cannot be proved. 3. All 157 tests under `tests/bugbash/sandbox/` pass on Bun 1.3.12. 4. I ran mutation checks on 6 rules. Five were caught: the job-name check, the boot and PID-namespace check, the start-time check, the exit-3 count and the launch listing. The sixth, a label check inside `listJobs()`, survived because it was redundant: the `docker ps` label filter already selects the checkout, and `removeJob()` checks the name and both labels again before any removal. I removed it. ## Risks Test tooling only. A wrong owner state could remove a container whose launcher still runs. That needs a container with this checkout's label and job name, on this boot and PID namespace, whose recorded PID start time no longer matches a running process. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$51.47`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=51.47 -->
1 parent 21521a3 commit 67da7c8

4 files changed

Lines changed: 408 additions & 17 deletions

File tree

‎Makefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ include fmt.mk
9191
.PHONY: all build dev start clean help
9292
.PHONY: build-renderer version build-icons build-static build-docker-runtime verify-docker-runtime-artifacts
9393
.PHONY: lint lint-fix typecheck static-check static-check-full
94-
.PHONY: test-bugbash-repros test-bugbash-known-failures bugbash-sandbox-key bugbash-sandbox-image bugbash-sandbox-publish
94+
.PHONY: test-bugbash-repros test-bugbash-known-failures bugbash-sandbox-key bugbash-sandbox-image bugbash-sandbox-publish bug-bash-sandbox-recover
9595
.PHONY: test test-unit test-unit-ci test-integration test-watch test-coverage test-e2e test-e2e-perf perf-tape-replay smoke-test
9696
.PHONY: dist dist-mac dist-win dist-linux install-mac-arm64 ensure-mac-sharp-runtime-deps check-appimage-icons check-mac-attach-file-runtime
9797
.PHONY: vscode-ext vscode-ext-install
@@ -235,6 +235,11 @@ bugbash-sandbox-image: ## Build the bug-bash sandbox image locally (linux/amd64)
235235
bugbash-sandbox-publish: ## Workflow only: build and push the bug-bash sandbox image, print its image.json record
236236
@./tests/bugbash/sandbox/build.sh --push
237237

238+
# Every sandbox launch lists the job containers that crashed launches of this checkout left
239+
# (#5882). This removes the ones whose launcher is gone, by ID, and lists the rest.
240+
bug-bash-sandbox-recover: ## Remove bug-bash sandbox containers of this checkout whose launcher is gone
241+
@cd tests/bugbash && bun sandbox/launch.ts --recover
242+
238243
# Bug-bash repro tests (tests/bugbash/repros/*.e2e.ts): exact UI steps, no model calls, against a
239244
# seeded `xum server` (tests/bugbash/startApp.ts). e2e needs Node.js 22.22.3+ or 24.8+ (E2E_NODE)
240245
# and its own Chromium: `node_modules/.bin/e2e-web install chromium`.

‎tests/bugbash/sandbox/launch.ts‎

Lines changed: 89 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
* Runs one exact-step bug-bash e2e job in a disposable container: the bug-bash sandbox (#5714).
33
* Usage, from tests/bugbash, with BUGBASH_AI=mock:
44
* bun sandbox/launch.ts -- run --config e2e.config.ts --output .e2e/<folder> [e2e args...]
5+
* bun sandbox/launch.ts --recover (make bug-bash-sandbox-recover; see recover())
56
*
67
* The container runs the e2e CLI, Chromium and the seeded app with the pinned image (runner.ts).
78
* It gets no network, no capabilities, a read-only root, copies of the git-listed inputs, and
@@ -29,6 +30,22 @@ const DEADLINE_MS = 30 * 60_000;
2930
const log = (message: string) => console.error(`sandbox ${message}`);
3031
const sha = (text: string) => crypto.createHash("sha256").update(text).digest("hex");
3132
const bootId = () => fs.readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim();
33+
const pidNamespace = () => fs.readlinkSync("/proc/self/ns/pid").replace(/\D/g, "");
34+
/** The checkout label: the first 12 hex of the sha256 of its real path. Job names hold 6. */
35+
const checkoutId = (root: string) => sha(root).slice(0, 12);
36+
/**
37+
* The fields of /proc/<pid>/stat after the command name: [0] the state, [19] the start time in
38+
* clock ticks. Undefined when the PID is gone.
39+
*/
40+
function procStat(pid: string): string[] | undefined {
41+
try {
42+
const stat = fs.readFileSync(`/proc/${pid}/stat`, "utf8");
43+
return stat.slice(stat.lastIndexOf(")") + 2).split(" ");
44+
} catch (error) {
45+
if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined;
46+
throw error;
47+
}
48+
}
3249

3350
/**
3451
* Only the mock app AI runs until the provider proxy (#5714): no provider key enters the
@@ -48,12 +65,71 @@ function bind(src: string, dst: string): string[] {
4865
return ["--mount", `type=bind,src=${src},dst=${dst},readonly`];
4966
}
5067

51-
/** boot id : PID namespace : PID : start time of this launcher. A later sweep reads it. */
68+
/** boot id : PID namespace : PID : start time of this launcher. ownerState() reads it. */
5269
function ownerLabel(): string {
53-
const pidns = fs.readlinkSync("/proc/self/ns/pid").replace(/\D/g, "");
54-
const stat = fs.readFileSync(`/proc/${process.pid}/stat`, "utf8");
55-
const start = stat.slice(stat.lastIndexOf(")") + 2).split(" ")[19];
56-
return `${bootId()}:${pidns}:${process.pid}:${start}`;
70+
return `${bootId()}:${pidNamespace()}:${process.pid}:${procStat(String(process.pid))?.[19]}`;
71+
}
72+
73+
/**
74+
* Whether the launcher named by an owner label still runs. "dead" needs this boot and this PID
75+
* namespace, and a PID that is gone, a zombie (it exited and waits for its parent), or now has
76+
* another start time (a reused PID). Anything that
77+
* cannot be checked from here (another boot, another namespace, a malformed label, an
78+
* unreadable /proc entry) is "cannot tell", and recover() leaves it.
79+
*/
80+
export function ownerState(owner: string): "dead" | "alive" | "cannot tell" {
81+
const parts = owner.split(":");
82+
if (parts.length !== 4 || !/^\d+$/.test(parts[2]) || !/^\d+$/.test(parts[3]))
83+
return "cannot tell";
84+
const [boot, pidns, pid, start] = parts;
85+
if (boot !== bootId() || pidns !== pidNamespace()) return "cannot tell";
86+
try {
87+
const stat = procStat(pid);
88+
return stat != null && !/^[ZX]$/.test(stat[0]) && stat[19] === start ? "alive" : "dead";
89+
} catch {
90+
return "cannot tell";
91+
}
92+
}
93+
94+
/**
95+
* `make bug-bash-sandbox-recover` (#5882): removes the job containers of this checkout whose
96+
* launcher is dead. It needs the job name pattern and a dead owner (ownerState), and it removes
97+
* by ID after the same name and label check as cleanup. It lists every other container and
98+
* leaves it. The owner label names a launcher, not the daemon, so this runs only on purpose,
99+
* and it prints the Docker endpoint first. Exit 3 when a removal cannot be proved.
100+
*/
101+
export async function recover(o: LaunchOptions & { log?: (line: string) => void }) {
102+
const say = o.log ?? log;
103+
const session = new Session(o.stop, { root: o.root, log: say });
104+
const checkout = checkoutId(o.root);
105+
const jobName = new RegExp(`^xbb-${checkout.slice(0, 6)}-[0-9a-f]{6}$`);
106+
let unproved = 0;
107+
let failure: { error: unknown } | undefined;
108+
try {
109+
say(`recover: docker endpoint ${await session.connect()}, checkout ${checkout}`);
110+
for (const { id, job } of await session.listJobs(checkout)) {
111+
const owner = ownerState(job.owner);
112+
const named = jobName.test(job.name);
113+
if (owner !== "dead" || !named) {
114+
say(`recover: left ${job.name}: owner ${owner}${named ? "" : ", not a job name"}`);
115+
continue;
116+
}
117+
// A removal that started finishes (cleanup commands ignore a stop); no new one starts.
118+
if (o.stop.aborted) throw new Stopped(String(o.stop.reason));
119+
const state = await session.removeJob(id, job);
120+
say(`recover: ${job.name} ${state}`);
121+
if (state !== "removed") unproved += 1;
122+
}
123+
} catch (error) {
124+
failure = { error };
125+
}
126+
const state = await session.cleanup();
127+
if (state.startsWith("unknown")) {
128+
say(`recover: cleanup ${state}`);
129+
return 3;
130+
}
131+
if (failure) throw failure.error;
132+
return unproved > 0 ? 3 : 0;
57133
}
58134

59135
export interface LaunchOptions {
@@ -86,12 +162,17 @@ export async function launch(args: string[], o: LaunchOptions): Promise<number>
86162
);
87163
mounts(); // before preflight, and again just before `docker run`
88164
const session = new Session(o.stop, { root: o.root });
89-
const checkout = sha(o.root).slice(0, 12);
165+
const checkout = checkoutId(o.root);
90166
const name = `xbb-${checkout.slice(0, 6)}-${crypto.randomBytes(3).toString("hex")}`;
91167
const jobDir = path.join(os.tmpdir(), "xum-bugbash-sandbox", checkout, name);
92168
let made = false;
93169

94170
const run = async (): Promise<number> => {
171+
// Leftovers of crashed launches (#5882) are only listed here, first, so a launch that fails
172+
// later (a long or failed pull) still shows them. recover removes the dead ones.
173+
await session.connect();
174+
for (const { job } of await session.listJobs(checkout))
175+
log(`leftover: ${job.name} owner ${ownerState(job.owner)} (make bug-bash-sandbox-recover)`);
95176
const image = await session.ensureImage();
96177
fs.mkdirSync(path.dirname(jobDir), { recursive: true, mode: 0o700 });
97178
fs.mkdirSync(jobDir, { mode: 0o700 }); // EEXIST: not this job's folder, so cleanup keeps it
@@ -164,7 +245,8 @@ if (import.meta.main) {
164245
const given = process.argv.slice(2);
165246
const args = given[0] === "--" ? given.slice(1) : given;
166247
const options = { root: ROOT, cwd: process.cwd(), env: process.env, stop: controller.signal };
167-
launch(args, options).then(
248+
const recovering = args.length === 1 && args[0] === "--recover";
249+
(recovering ? recover(options) : launch(args, options)).then(
168250
(code) => process.exit(code),
169251
(error: unknown) => {
170252
if (error instanceof Stopped) {

0 commit comments

Comments
 (0)