Skip to content

🤖 tests: list bug-bash sandbox crash leftovers and add a recover target - #6072

Merged
ThomasK33 merged 5 commits into
mainfrom
tests/5714-c2-sandbox-recover
Oct 10, 2026
Merged

ThomasK33 merged 5 commits into
mainfrom
tests/5714-c2-sandbox-recover

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Summary

Lists the job containers that crashed launches left behind, and adds make bug-bash-sandbox-recover to remove the ones whose launcher is gone. This is PR C2 of the plan to run the model-driven bug bash in the sandbox, stacked on #6069 (C). It must land before B1, which activates model-driven jobs.

Refs #5714
Fixes #5882

What changes

  1. Every launch lists the job containers of its checkout, found by the xum.bugbash.checkout label, right after docker info. Each line shows the owner state: dead, alive or cannot tell. The launch removes nothing.
  2. make bug-bash-sandbox-recover (launch.ts --recover) prints the Docker endpoint first. It removes a container only when both hold:
    • Its name matches this checkout's job pattern ^xbb-<checkout6>-[0-9a-f]{6}$.
    • Its owner is dead: the same boot ID and PID namespace, and a PID that is gone or now has another start time (a reused PID).
  3. Each removal is by ID, after the same name and label check that cleanup uses (🤖 tests: clean up the bug-bash sandbox container by its cidfile ID #6069). Every other container is listed and left. A removal it cannot prove exits 3.
  4. The owner label names the launcher, not the Docker daemon, so removal happens only when a person runs the target on purpose. That is why the target prints the endpoint before it removes anything.

Real-Docker dogfood (Docker 27.5.1)

I created four test containers with names that only this check uses, then ran a launch (contract case (a) of #6069) and the recover target twice.

Container Owner Launch listing First recover Second recover (after the owner ended)
xbb-0b6a51-c2dead dead owner dead removed by ID (already gone)
xbb-0b6a51-c2a11e a live sleep owner alive left removed
xbb-0b6a51-c2live the same live sleep owner alive left: not a job name left: owner dead, not a job name
xbbprobe-c2-other dead, other checkout not listed not listed not listed

The launch still ended cleanup: removed for its own job container, and both recover runs exited 0. I removed the two remaining test containers by their exact IDs afterwards.

1-launch.log
case a: launcher start 2026-10-10T15:39:09.174Z checkout=0b6a51dd309e
+1.74s sandbox leftover: xbb-0b6a51-c2a11e owner alive (make bug-bash-sandbox-recover)
+1.74s sandbox leftover: xbb-0b6a51-c2live owner alive (make bug-bash-sandbox-recover)
+1.75s sandbox leftover: xbb-0b6a51-c2dead owner dead (make bug-bash-sandbox-recover)
+2.11s sandbox xbb-0b6a51-28cbc7 staged 3414 files
+2.11s sandbox xbb-0b6a51-28cbc7 starts: --network none, mock app AI
+2.11s ACTION: SIGTERM to the launcher right after the spawn of docker run
+2.71s sandbox entry: the launcher is gone, so the job stops
+2.79s sandbox xbb-0b6a51-28cbc7 exit 137, 0 files, incomplete: no end frame
+2.79s sandbox xbb-0b6a51-28cbc7 cleanup: container fcd31d124888, ID from the cidfile
+3.03s sandbox xbb-0b6a51-28cbc7 cleanup: removed
+3.03s sandbox stopped: SIGTERM
+3.03s launcher exit 143
containers of this checkout after exit: 2dc93aaaf0e1 xbb-0b6a51-c2a11e Created
bcefb43a91bc xbb-0b6a51-c2live Created
d1edf4985d9f xbb-0b6a51-c2dead Created
export files: 0
2-recover.log
$ make bug-bash-sandbox-recover
sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e
sandbox recover: left xbb-0b6a51-c2a11e: owner alive
sandbox recover: left xbb-0b6a51-c2live: owner alive, not a job name
sandbox xbb-0b6a51-c2dead cleanup: container d1edf4985d9f, ID from the leftover list
sandbox recover: xbb-0b6a51-c2dead removed
exit 0
4-recover-again.log
$ make bug-bash-sandbox-recover   (after the live owner ended)
sandbox recover: docker endpoint unix:///var/run/docker.sock, checkout 0b6a51dd309e
sandbox xbb-0b6a51-c2a11e cleanup: container 2dc93aaaf0e1, ID from the leftover list
sandbox recover: xbb-0b6a51-c2a11e removed
sandbox recover: left xbb-0b6a51-c2live: owner dead, not a job name
exit 0

Validation

  1. The tests ran failing-first: the test file failed to load, because launch.ts had no ownerState or recover export.
  2. New tests: ownerState (dead, alive, a reused PID, another boot, another PID namespace, malformed labels); a launch lists every leftover and removes nothing; recover removes only dead owners with a job name, by ID, and pulls or runs nothing; recover exits 3 when a removal cannot be proved.
  3. All 157 tests under tests/bugbash/sandbox/ pass on Bun 1.3.12.
  4. I ran mutation checks on 6 rules. Five were caught: the job-name check, the boot and PID-namespace check, the start-time check, the exit-3 count and the launch listing. The sixth, a label check inside listJobs(), survived because it was redundant: the docker ps label filter already selects the checkout, and removeJob() checks the name and both labels again before any removal. I removed it.

Risks

Test tooling only. A wrong owner state could remove a container whose launcher still runs. That needs a container with this checkout's label and job name, on this boot and PID namespace, whose recorded PID start time no longer matches a running process.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $51.47

@ThomasK33
ThomasK33 added this pull request to stack #6070 October 10, 2026 15:42
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T17:24:40.739560Z 2bd7b63 Manual request
🔒 Security Review ✅ Completed 2026-10-10T17:24:11.315829Z 2bd7b63 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: cc326ab1bf

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc326ab1bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/bugbash/sandbox/launch.ts Outdated
Comment thread tests/bugbash/sandbox/runner.ts Outdated
Comment thread tests/bugbash/sandbox/launch.ts
Comment thread tests/bugbash/sandbox/launch.ts Outdated
Comment thread tests/bugbash/sandbox/launch.ts Outdated
@ThomasK33
ThomasK33 force-pushed the tests/5714-c2-sandbox-recover branch from cc326ab to fa63d84 Compare October 10, 2026 16:02
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 62c1a95133

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62c1a95133

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/bugbash/sandbox/runner.test.ts Outdated
@ThomasK33
ThomasK33 force-pushed the tests/5714-c2-sandbox-recover branch from 62c1a95 to 57e3608 Compare October 10, 2026 16:43
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 57e3608887

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 57e3608887

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/bugbash/sandbox/runner.test.ts Outdated
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: d9faf631db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: d9faf631db

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Base automatically changed from tests/5714-c-runner-cleanup to main October 10, 2026 17:00
…er target

PR C2 of the #5714 sandbox plan (#5882, plan item 11), stacked on PR C.

- Every launch lists the job containers of its checkout (by the checkout label) after
  `docker info`, with the owner state dead, alive or cannot tell. It removes nothing.
- `make bug-bash-sandbox-recover` (`launch.ts --recover`) prints the Docker endpoint,
  then removes only containers whose name matches this checkout's job pattern and whose
  owner is dead: same boot and PID namespace, and a PID that is gone or has another start
  time. Removal is by ID, after the same name and label check as cleanup. Everything else
  is listed and left. Exit 3 when a removal cannot be proved.

---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$46.10`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=46.10 -->
- A launch lists the leftovers right after the Docker preflight, before it prepares the
  image, so a launch whose pull fails still shows them.
- listJobs() ignores a container only when inspect says "No such container"; any other
  inspect failure refuses, so recover cannot exit 0 past an unseen container.
- recover() checks the stop before each removal: a removal that started finishes, and no
  new one starts.
- ownerState() counts a zombie launcher (state Z or X) as dead.
- recover() returns 3 when the session cleanup state is unknown.

---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$52.06`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=52.06 -->
---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$52.06`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=52.06 -->
Codex review round 2: a child that exits at once can be reaped by the shell before the
shell execs `sleep 30`, and the test then waited forever. The child now exits 300 ms later,
after the exec, and the wait is bounded with a clear assertion.

---

_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$52.06`_

<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=52.06 -->
- runner.test.ts: the zombie-launcher test now releases the child through a FIFO
  only after the parent shell became `sleep 30`, instead of a fixed `sleep 0.3`
  that a descheduled shell could lose (Codex r3).
- runner.ts: #endJob waits at most 10 s for each onStop hook, then logs and
  closes the lifeline anyway, so a hung hook (B1's proxy close) cannot block
  cleanup. Test: a never-settling hook still lets stdin close and cleanup finish.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2bd7b633a9

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Oct 10, 2026
@ThomasK33

Copy link
Copy Markdown
Member Author

Merge note: the maintainer approved a narrow exception to the "Codex approval on the final head" rule for this PR only.

  • Codex approved head d9faf631db ("Didn't find any major issues" plus a +1).
  • GitHub then restacked the PR onto main as 2bd7b633a9. The C2 patch is byte-identical: git diff 476576afae d9faf631db and git diff <main merge-base> 2bd7b633a9 have the same sha256 (c89f5487…).
  • On 2bd7b633a9, Codex completed both review lanes with no comments. Its security review found no issues, but it added no +1. One re-request was silent too.
  • A clean-context readiness review said ready, 0 threads are open, and Required passes.

The exception covers this unchanged patch only. It is not a general rule.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

Merged via the queue into main with commit 67da7c8 Oct 10, 2026
32 of 33 checks passed
@ThomasK33
ThomasK33 deleted the tests/5714-c2-sandbox-recover branch October 10, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🤖 tests: recover bug-bash sandbox containers left by a launcher crash

1 participant