Skip to content

Commit abfa4e3

Browse files
authored
🤖 tests: runner library for the bug-bash sandbox, with the first image digest (#5875)
## Summary This is PR 2 of 4 for the approved B1 plan (#5714). It adds the runner library of the bug-bash sandbox, plus `image.json` with the first published digest. The library has no entry point, so nothing can run a job yet. PR 3 adds staging, the mount checks, the app log and the launch command. ## Background #5818 added the image workflow, and a manual publish ran on `main`: [run 37759827317](https://github.com/coder/xum/actions/runs/37759827317) (`workflow_dispatch`, `main`, `752a624e15`). This PR records that digest in a reviewed file. That is the "a person opens the digest PR" step of the plan. It also adds the code that uses the digest. ## Implementation 1. `tests/bugbash/sandbox/image.json` pins the published image. I copied every value from tool output, not by hand: - The digest comes from the publish run's Attest step. It equals the sha256 of the registry manifest. - The key and the playwright-core version come from the image's labels. - `publishedFrom` is the run's head SHA. - The key equals `make bugbash-sandbox-key` on this branch. 2. `tests/bugbash/sandbox/runner.ts` has two parts. `readImageLock()` validates `image.json`. `Session` does the work: - **ensureImage()** first compares the checkout's inputs key (`build.sh --key`) with `image.json`. A mismatch refuses as a stale image, before any docker command. Next it finds the local daemon. If the image is missing, it pulls only `name@digest`. Then it checks that the image label `org.xum.bugbash.inputs` equals the key. An empty `docker images` list with exit 0 means "absent". Any query failure refuses, so a failure is never read as "absent". The runner never builds. - **Docker preflight** comes from #5802. It refuses non-Linux hosts, root, remote or ssh endpoints, Docker Desktop and rootless Docker. After `docker context inspect`, every docker command gets only `PATH`, `DOCKER_HOST` and an empty private `DOCKER_CONFIG`. - **Children:** every command is an async child that the session tracks. The entry point (PR 3) will abort the session's `AbortSignal` from its SIGINT and SIGTERM handlers. Then running children get SIGTERM, and SIGKILL after 5 s. Job commands started after that refuse with `Stopped`. - **cleanup(job)** is one idempotent promise for success, error and stop. It waits for every child. It removes only the container that matches the name and both labels (owner and checkout), and it confirms the removal. It reports `unknown: …`, never success, when it cannot read the container state. It removes the private client folder. It never removes an image. ## Validation 1. Gate, the anonymous pull: in a fresh `docker:27-dind` with no client config and no images, `docker pull ghcr.io/coder/xum-bugbash-sandbox@sha256:61adf1a543f2b1cc1373506556eefad52176aabd5bb1e491927eef82bde8fa73` exited 0 in 30,228 ms and gave 1,425,818,783 bytes. An anonymous `imagetools inspect` (empty `DOCKER_CONFIG`) gave the same digest, so the package is public. 2. Gate, the stale-key refusal: `runner.test.ts` "a stale inputs key refuses before any docker command". 3. `runner.test.ts` runs the real `build.sh` in throwaway git repos with a fake `docker`. It covers: - a pull by digest when the image is missing, and no pull when it is present; - a refusal on a label mismatch, on an image query failure, on a remote endpoint, with no daemon, on Docker Desktop and on rootless Docker; - the private client env; - a stop during a pull that ignores SIGTERM: it ends with `Stopped` after SIGKILL, new commands refuse, and cleanup still removes the job container with the full filter; - cleanup that reports `unknown` when `ps` fails, and that returns the same promise on a second call. 4. Mutation check: 10 mutants, all fail the tests. They are: no stale-key check, query failure read as absent, no label check, no SIGKILL escalation, cleanup gated by the stop, unknown reported as removed, user env passed to docker, cleanup not idempotent, remote endpoint allowed, and the checkout label filter dropped. 5. `make static-check` passes. `bun test ./tests/bugbash` passes (55 tests). Size: 3 files, +454 lines with tests. ## Risks Low. This PR adds test tooling only, and nothing calls the library yet. No file under `src/` changes. The #5815 host pause stays. Refs #5714 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$111.52`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=111.52 -->
1 parent 752a624 commit abfa4e3

3 files changed

Lines changed: 454 additions & 0 deletions

File tree

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"image": "ghcr.io/coder/xum-bugbash-sandbox",
3+
"digest": "sha256:61adf1a543f2b1cc1373506556eefad52176aabd5bb1e491927eef82bde8fa73",
4+
"inputsKey": "0d15da9c1a6a87051e21b589933ffefb000af794de69d2741661fded30e7abf7",
5+
"playwrightCore": "1.63.0",
6+
"publishedFrom": "752a624e15e0a332763d6fef8329d86d9faebe36"
7+
}
Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
import { afterEach, beforeEach, expect, test } from "bun:test";
2+
import { spawnSync } from "node:child_process";
3+
import * as fs from "node:fs";
4+
import * as os from "node:os";
5+
import * as path from "node:path";
6+
import { readImageLock, Refusal, Session, Stopped } from "./runner";
7+
8+
// Each test runs the real build.sh in a throwaway git repo and a fake `docker` on PATH. The
9+
// runner gives docker a stripped env, so the fake reads its answers from bin/fake.env.
10+
const SANDBOX = "tests/bugbash/sandbox";
11+
const DIGEST = `sha256:${"61".repeat(32)}`;
12+
const REF = `ghcr.io/coder/xum-bugbash-sandbox@${DIGEST}`;
13+
const FAKE = `#!/bin/sh
14+
bin=$(dirname "$0"); . "$bin/fake.env"
15+
echo "$* [home=\${HOME-} cfg=\${DOCKER_CONFIG-}]" >> "$bin/calls.log"
16+
case "$1" in
17+
context) echo "$HOST" ;;
18+
info) echo "$INFO" ;;
19+
images) [ "$IMAGES_RC" = 0 ] || { echo "daemon down" >&2; exit 1; }; echo "$IMAGES" ;;
20+
pull) if [ "$PULL" = hang ]; then trap '' TERM; exec sleep 30; fi ;;
21+
image) echo "{\\"org.xum.bugbash.inputs\\":\\"$LABEL\\"}" ;;
22+
ps) [ "$PS_RC" = 0 ] || exit 1; cat "$bin/containers" ;;
23+
rm) : > "$bin/containers" ;;
24+
*) exit 9 ;;
25+
esac
26+
`;
27+
28+
let root = "";
29+
let bin = "";
30+
let key = "";
31+
const savedPath = process.env.PATH;
32+
beforeEach(() => {
33+
root = fs.mkdtempSync(path.join(os.tmpdir(), "xbb-runner-"));
34+
fs.mkdirSync(path.join(root, SANDBOX), { recursive: true });
35+
for (const name of ["build.sh", "Dockerfile"])
36+
fs.copyFileSync(path.join(import.meta.dir, name), path.join(root, SANDBOX, name));
37+
fs.writeFileSync(
38+
path.join(root, "bun.lock"),
39+
'"@e2e-dev/web/playwright-core": ["playwright-core@1.63.0", ""],\n'
40+
);
41+
for (const args of [
42+
["init", "-q"],
43+
["add", "-A"],
44+
["-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", "x"],
45+
])
46+
expect(spawnSync("git", args, { cwd: root }).status).toBe(0);
47+
key = spawnSync(path.join(root, SANDBOX, "build.sh"), ["--key"], {
48+
encoding: "utf8",
49+
}).stdout.trim();
50+
expect(key).toMatch(/^[0-9a-f]{64}$/);
51+
lock(key);
52+
bin = path.join(root, "bin");
53+
fs.mkdirSync(bin);
54+
fs.writeFileSync(path.join(bin, "docker"), FAKE, { mode: 0o755 });
55+
fs.writeFileSync(path.join(bin, "containers"), "");
56+
fake();
57+
process.env.PATH = `${bin}:${savedPath ?? ""}`;
58+
});
59+
afterEach(async () => {
60+
await Promise.all(sessions.splice(0).map((s) => s.cleanup()));
61+
process.env.PATH = savedPath;
62+
fs.rmSync(root, { recursive: true, force: true });
63+
});
64+
65+
function lock(inputsKey: string, digest = DIGEST) {
66+
const record = {
67+
image: "ghcr.io/coder/xum-bugbash-sandbox",
68+
digest,
69+
inputsKey,
70+
playwrightCore: "1.63.0",
71+
publishedFrom: "7".repeat(40),
72+
};
73+
fs.writeFileSync(path.join(root, SANDBOX, "image.json"), JSON.stringify(record));
74+
}
75+
function fake(over: Record<string, string> = {}) {
76+
const vars = {
77+
HOST: "unix:///var/run/docker.sock",
78+
INFO: '{"OSType":"linux","OperatingSystem":"Ubuntu 22.04","SecurityOptions":["name=seccomp"]}',
79+
IMAGES_RC: "0",
80+
IMAGES: "",
81+
PULL: "ok",
82+
LABEL: key,
83+
PS_RC: "0",
84+
...over,
85+
};
86+
fs.writeFileSync(
87+
path.join(bin, "fake.env"),
88+
Object.entries(vars)
89+
.map(([k, v]) => `${k}='${v}'\n`)
90+
.join("")
91+
);
92+
}
93+
const calls = () =>
94+
fs.existsSync(path.join(bin, "calls.log"))
95+
? fs.readFileSync(path.join(bin, "calls.log"), "utf8")
96+
: "";
97+
/** The error a promise rejects with (`expect().rejects` is not typed as awaitable here). */
98+
const failure = (p: Promise<unknown>) =>
99+
p.then(
100+
() => () => undefined,
101+
(e: unknown) => () => {
102+
throw e;
103+
}
104+
);
105+
// Every session gets cleaned up, so no private client folder stays behind.
106+
const sessions: Session[] = [];
107+
const session = (stop = new AbortController()) => {
108+
const s = new Session(stop.signal, { root });
109+
sessions.push(s);
110+
return s;
111+
};
112+
113+
test("the committed image.json is a valid lock, and a malformed one refuses", () => {
114+
expect(readImageLock().digest).toMatch(/^sha256:[0-9a-f]{64}$/);
115+
lock(key, "sha256:short");
116+
expect(() => readImageLock(root)).toThrow(Refusal);
117+
});
118+
119+
test("a stale inputs key refuses before any docker command", async () => {
120+
lock("0".repeat(64));
121+
expect(await failure(session().ensureImage())).toThrow(/stale image/);
122+
await Bun.sleep(0);
123+
expect(calls()).toBe("");
124+
});
125+
126+
test("a missing image is pulled by digest, with a private client config", async () => {
127+
const s = session();
128+
expect(await s.ensureImage()).toBe(REF);
129+
const lines = calls().trim().split("\n");
130+
expect(lines.map((l) => l.split(" ")[0])).toEqual(["context", "info", "images", "pull", "image"]);
131+
expect(lines[3]).toStartWith(`pull --quiet ${REF} `);
132+
// After `context inspect`, docker gets no HOME and an empty config folder that cleanup removes.
133+
const cfg = /cfg=(\S+)\]/.exec(lines[2])![1];
134+
expect(lines[2]).toContain("[home= ");
135+
expect(fs.readdirSync(cfg)).toEqual([]);
136+
expect(await s.cleanup()).toBe("removed");
137+
expect(fs.existsSync(cfg)).toBe(false);
138+
});
139+
140+
test("a present image is not pulled", async () => {
141+
fake({ IMAGES: "sha256:abc" });
142+
expect(await session().ensureImage()).toBe(REF);
143+
expect(calls()).not.toContain("pull");
144+
});
145+
146+
test.each([
147+
[
148+
"a label for other inputs",
149+
{ IMAGES: "sha256:abc", LABEL: "f".repeat(64) },
150+
/label org.xum.bugbash.inputs/,
151+
],
152+
["an image query failure (not 'absent')", { IMAGES_RC: "1" }, /docker images: daemon down/],
153+
["a remote endpoint", { HOST: "tcp://10.0.0.1:2375" }, /not a local socket/],
154+
["no daemon", { INFO: '{"OSType":""}' }, /no daemon answered/],
155+
[
156+
"Docker Desktop",
157+
{ INFO: '{"OSType":"linux","OperatingSystem":"Docker Desktop"}' },
158+
/Docker Desktop/,
159+
],
160+
[
161+
"rootless Docker",
162+
{ INFO: '{"OSType":"linux","SecurityOptions":["name=rootless"]}' },
163+
/rootless/,
164+
],
165+
])("%s refuses, and nothing is pulled", async (_name, over, message) => {
166+
fake(over);
167+
expect(await failure(session().ensureImage())).toThrow(message);
168+
await Bun.sleep(50);
169+
expect(calls()).not.toContain("pull");
170+
});
171+
172+
test("a stop ends a running pull; cleanup still runs and removes only the job's container", async () => {
173+
fake({ PULL: "hang" });
174+
fs.writeFileSync(path.join(bin, "containers"), "c1\n");
175+
const stop = new AbortController();
176+
const s = session(stop);
177+
const pending = s.ensureImage();
178+
while (!calls().includes("pull ")) await Bun.sleep(20);
179+
const stoppedAt = Date.now();
180+
stop.abort("SIGTERM");
181+
// The fake pull ignores SIGTERM, so it ends only by SIGKILL after the grace period.
182+
expect(await pending.catch((e: unknown) => e)).toEqual(new Stopped("SIGTERM"));
183+
expect(Date.now() - stoppedAt).toBeGreaterThanOrEqual(4_500);
184+
expect(await failure(s.ensureImage())).toThrow(Stopped);
185+
186+
const job = { name: "xbb-1", owner: "boot:pid", checkout: "abc" };
187+
const first = s.cleanup(job);
188+
expect(s.cleanup(job)).toBe(first);
189+
expect(await first).toBe("removed");
190+
const ps = calls()
191+
.split("\n")
192+
.find((l) => l.startsWith("ps "))!;
193+
expect(ps).toContain(
194+
"--filter name=^/xbb-1$ --filter label=xum.bugbash.owner=boot:pid --filter label=xum.bugbash.checkout=abc"
195+
);
196+
expect(calls()).toContain("rm -f c1 ");
197+
}, 15_000);
198+
199+
test("cleanup reports an unknown container state, never success", async () => {
200+
fake({ IMAGES: "sha256:abc", PS_RC: "1" });
201+
const s = session();
202+
await s.ensureImage();
203+
expect(await s.cleanup({ name: "xbb-1", owner: "o", checkout: "c" })).toStartWith("unknown: ");
204+
});

0 commit comments

Comments
 (0)