Repository navigation
Commit abfa4e3
authored
🤖 tests: runner library for the bug-bash sandbox, with the first image digest (#5875)
## Summary
This is PR 2 of 4 for the approved B1 plan (#5714). It adds the runner
library of the bug-bash sandbox, plus `image.json` with the first
published digest. The library has no entry point, so nothing can run a
job yet. PR 3 adds staging, the mount checks, the app log and the launch
command.
## Background
#5818 added the image workflow, and a manual publish ran on `main`: [run
37759827317](https://github.com/coder/xum/actions/runs/37759827317)
(`workflow_dispatch`, `main`, `752a624e15`). This PR records that digest
in a reviewed file. That is the "a person opens the digest PR" step of
the plan. It also adds the code that uses the digest.
## Implementation
1. `tests/bugbash/sandbox/image.json` pins the published image. I copied
every value from tool output, not by hand:
- The digest comes from the publish run's Attest step. It equals the
sha256 of the registry manifest.
- The key and the playwright-core version come from the image's labels.
- `publishedFrom` is the run's head SHA.
- The key equals `make bugbash-sandbox-key` on this branch.
2. `tests/bugbash/sandbox/runner.ts` has two parts. `readImageLock()`
validates `image.json`. `Session` does the work:
- **ensureImage()** first compares the checkout's inputs key (`build.sh
--key`) with `image.json`. A mismatch refuses as a stale image, before
any docker command. Next it finds the local daemon. If the image is
missing, it pulls only `name@digest`. Then it checks that the image
label `org.xum.bugbash.inputs` equals the key. An empty `docker images`
list with exit 0 means "absent". Any query failure refuses, so a failure
is never read as "absent". The runner never builds.
- **Docker preflight** comes from #5802. It refuses non-Linux hosts,
root, remote or ssh endpoints, Docker Desktop and rootless Docker. After
`docker context inspect`, every docker command gets only `PATH`,
`DOCKER_HOST` and an empty private `DOCKER_CONFIG`.
- **Children:** every command is an async child that the session tracks.
The entry point (PR 3) will abort the session's `AbortSignal` from its
SIGINT and SIGTERM handlers. Then running children get SIGTERM, and
SIGKILL after 5 s. Job commands started after that refuse with
`Stopped`.
- **cleanup(job)** is one idempotent promise for success, error and
stop. It waits for every child. It removes only the container that
matches the name and both labels (owner and checkout), and it confirms
the removal. It reports `unknown: …`, never success, when it cannot read
the container state. It removes the private client folder. It never
removes an image.
## Validation
1. Gate, the anonymous pull: in a fresh `docker:27-dind` with no client
config and no images, `docker pull
ghcr.io/coder/xum-bugbash-sandbox@sha256:61adf1a543f2b1cc1373506556eefad52176aabd5bb1e491927eef82bde8fa73`
exited 0 in 30,228 ms and gave 1,425,818,783 bytes. An anonymous
`imagetools inspect` (empty `DOCKER_CONFIG`) gave the same digest, so
the package is public.
2. Gate, the stale-key refusal: `runner.test.ts` "a stale inputs key
refuses before any docker command".
3. `runner.test.ts` runs the real `build.sh` in throwaway git repos with
a fake `docker`. It covers:
- a pull by digest when the image is missing, and no pull when it is
present;
- a refusal on a label mismatch, on an image query failure, on a remote
endpoint, with no daemon, on Docker Desktop and on rootless Docker;
- the private client env;
- a stop during a pull that ignores SIGTERM: it ends with `Stopped`
after SIGKILL, new commands refuse, and cleanup still removes the job
container with the full filter;
- cleanup that reports `unknown` when `ps` fails, and that returns the
same promise on a second call.
4. Mutation check: 10 mutants, all fail the tests. They are: no
stale-key check, query failure read as absent, no label check, no
SIGKILL escalation, cleanup gated by the stop, unknown reported as
removed, user env passed to docker, cleanup not idempotent, remote
endpoint allowed, and the checkout label filter dropped.
5. `make static-check` passes. `bun test ./tests/bugbash` passes (55
tests).
Size: 3 files, +454 lines with tests.
## Risks
Low. This PR adds test tooling only, and nothing calls the library yet.
No file under `src/` changes. The #5815 host pause stays.
Refs #5714
---
_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high` • Cost: `$111.52`_
<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs=111.52 -->1 parent 752a624 commit abfa4e3
3 files changed
Lines changed: 454 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
0 commit comments