Repository navigation
Commit 67da7c8
authored
🤖 tests: list bug-bash sandbox crash leftovers and add a recover target (#6072)
## Summary
Lists the job containers that crashed launches left behind, and adds
`make bug-bash-sandbox-recover` to remove the ones whose launcher is
gone. This is PR C2 of the plan to run the model-driven bug bash in the
sandbox, stacked on #6069 (C). It must land before B1, which activates
model-driven jobs.
Refs #5714
Fixes #5882
## What changes
1. Every launch lists the job containers of its checkout, found by the
`xum.bugbash.checkout` label, right after `docker info`. Each line shows
the owner state: `dead`, `alive` or `cannot tell`. The launch removes
nothing.
2. `make bug-bash-sandbox-recover` (`launch.ts --recover`) prints the
Docker endpoint first. It removes a container only when both hold:
- Its name matches this checkout's job pattern
`^xbb-<checkout6>-[0-9a-f]{6}$`.
- Its owner is dead: the same boot ID and PID namespace, and a PID that
is gone or now has another start time (a reused PID).
3. Each removal is by ID, after the same name and label check that
cleanup uses (#6069). Every other container is listed and left. A
removal it cannot prove exits 3.
4. The owner label names the launcher, not the Docker daemon, so removal
happens only when a person runs the target on purpose. That is why the
target prints the endpoint before it removes anything.
## Real-Docker dogfood (Docker 27.5.1)
I created four test containers with names that only this check uses,
then ran a launch (contract case (a) of #6069) and the recover target
twice.
| Container | Owner | Launch listing | First recover | Second recover
(after the owner ended) |
| --- | --- | --- | --- | --- |
| `xbb-0b6a51-c2dead` | dead | `owner dead` | removed by ID | (already
gone) |
| `xbb-0b6a51-c2a11e` | a live `sleep` | `owner alive` | left | removed
|
| `xbb-0b6a51-c2live` | the same live `sleep` | `owner alive` | left:
not a job name | left: owner dead, not a job name |
| `xbbprobe-c2-other` | dead, other checkout | not listed | not listed |
not listed |
The launch still ended `cleanup: removed` for its own job container, and
both recover runs exited 0. I removed the two remaining test containers
by their exact IDs afterwards.
<details>
<summary>1-launch.log</summary>
```text
case a: launcher start 2026-10-10T15:39:09.174Z checkout=0b6a51dd309e
+1.74s sandbox leftover: xbb-0b6a51-c2a11e owner alive (make bug-bash-sandbox-recover)
+1.74s sandbox leftover: xbb-0b6a51-c2live owner alive (make bug-bash-sandbox-recover)
+1.75s sandbox leftover: xbb-0b6a51-c2dead owner dead (make bug-bash-sandbox-recover)
+2.11s sandbox xbb-0b6a51-28cbc7 staged 3414 files
+2.11s sandbox xbb-0b6a51-28cbc7 starts: --network none, mock app AI
+2.11s ACTION: SIGTERM to the launcher right after the spawn of docker run
+2.71s sandbox entry: the launcher is gone, so the job stops
+2.79s sandbox xbb-0b6a51-28cbc7 exit 137, 0 files, incomplete: no end frame
+2.79s sandbox xbb-0b6a51-28cbc7 cleanup: container fcd31d124888, ID from the cidfile
+3.03s sandbox xbb-0b6a51-28cbc7 cleanup: removed
+3.03s sandbox stopped: SIGTERM
+3.03s launcher exit 143
containers of this checkout after exit: 2dc93aaaf0e1 xbb-0b6a51-c2a11e Created
bcefb43a91bc xbb-0b6a51-c2live Created
d1edf4985d9f xbb-0b6a51-c2dead Created
export files: 0
```
</details>
<details>
<summary>2-recover.log</summary>
```text
$ make bug-bash-sandbox-recover
sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e
sandbox recover: left xbb-0b6a51-c2a11e: owner alive
sandbox recover: left xbb-0b6a51-c2live: owner alive, not a job name
sandbox xbb-0b6a51-c2dead cleanup: container d1edf4985d9f, ID from the leftover list
sandbox recover: xbb-0b6a51-c2dead removed
exit 0
```
</details>
<details>
<summary>4-recover-again.log</summary>
```text
$ make bug-bash-sandbox-recover (after the live owner ended)
sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e
sandbox xbb-0b6a51-c2a11e cleanup: container 2dc93aaaf0e1, ID from the leftover list
sandbox recover: xbb-0b6a51-c2a11e removed
sandbox recover: left xbb-0b6a51-c2live: owner dead, not a job name
exit 0
```
</details>
## Validation
1. The tests ran failing-first: the test file failed to load, because
`launch.ts` had no `ownerState` or `recover` export.
2. New tests: `ownerState` (dead, alive, a reused PID, another boot,
another PID namespace, malformed labels); a launch lists every leftover
and removes nothing; recover removes only dead owners with a job name,
by ID, and pulls or runs nothing; recover exits 3 when a removal cannot
be proved.
3. All 157 tests under `tests/bugbash/sandbox/` pass on Bun 1.3.12.
4. I ran mutation checks on 6 rules. Five were caught: the job-name
check, the boot and PID-namespace check, the start-time check, the
exit-3 count and the launch listing. The sixth, a label check inside
`listJobs()`, survived because it was redundant: the `docker ps` label
filter already selects the checkout, and `removeJob()` checks the name
and both labels again before any removal. I removed it.
## Risks
Test tooling only. A wrong owner state could remove a container whose
launcher still runs. That needs a container with this checkout's label
and job name, on this boot and PID namespace, whose recorded PID start
time no longer matches a running process.
---
_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high` • Cost: `$51.47`_
<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs=51.47 -->1 parent 21521a3 commit 67da7c8
4 files changed
Lines changed: 408 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
| 94 | + | |
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| |||
235 | 235 | | |
236 | 236 | | |
237 | 237 | | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
238 | 243 | | |
239 | 244 | | |
240 | 245 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
32 | 49 | | |
33 | 50 | | |
34 | 51 | | |
| |||
48 | 65 | | |
49 | 66 | | |
50 | 67 | | |
51 | | - | |
| 68 | + | |
52 | 69 | | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
57 | 133 | | |
58 | 134 | | |
59 | 135 | | |
| |||
86 | 162 | | |
87 | 163 | | |
88 | 164 | | |
89 | | - | |
| 165 | + | |
90 | 166 | | |
91 | 167 | | |
92 | 168 | | |
93 | 169 | | |
94 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
95 | 176 | | |
96 | 177 | | |
97 | 178 | | |
| |||
164 | 245 | | |
165 | 246 | | |
166 | 247 | | |
167 | | - | |
| 248 | + | |
| 249 | + | |
168 | 250 | | |
169 | 251 | | |
170 | 252 | | |
| |||
0 commit comments