Skip to content

fix: Exit Cleanly When Native Executor Shuts Down Concurrently - #191

Merged
danny-avila merged 2 commits into
mainfrom
fix/native-executor-clean-shutdown
Sep 13, 2026
Merged

danny-avila merged 2 commits into
mainfrom
fix/native-executor-clean-shutdown

Conversation

@danny-avila

@danny-avila danny-avila commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

A native BYOM worker running under systemd with KillMode=control-group exited with status 1 on an ordinary systemctl stop or restart, even while idle, and the journal recorded Native executor is unavailable. The worker restarted and reconnected fine, so every deploy looked like a service failure without being one. Closes #190.

What raced. systemd delivers SIGTERM to the parent CLI and the forked SRT executor at the same moment. The child marks itself shutting down, begins closing its SRT sandbox, and ignores further IPC. The parent aborts the worker loop and reaches nativeCommandSandbox.close() in the CLI finally almost immediately. If the child is still connected at that point, stop() sends the close RPC, the child drops it, and when the child finally exits the lost handler rejects the pending RPC with Native executor is unavailable. That rejection propagated through close(), the CLI finally, and main().catch() into exit status 1. The same sequence applies to Ctrl-C and any supervisor that signals the whole process group.

  • Tolerate a lost, refused, or stalled close handshake in NativeProcessWorkspaceCommandSandbox.stop(). The child is terminated in finally whether or not it replies, and stop() has already drained any active command, so those cases carry no mutation ambiguity. A negative close reply from the executor, such as a failed SRT reset or scratch cleanup, is a real failure and still rejects so NativeWorkspaceCommandPool.close() can aggregate it. Quarantine decisions live on the execute path and are unchanged.
  • Report exit status 0 from the executor child when its own SRT teardown succeeds on a signal-driven shutdown, and keep status 1 for a failed or timed-out teardown. The parent never inspected this code, so this only makes the journal honest.

Tests

  • New: close() resolves when the child exits during the close handshake without replying. This fails on main with the same unavailable rejection seen in the issue.
  • New: close() skips the handshake entirely once the child is already lost, and later execute() calls remain fenced.
  • New: close() still rejects with the executor's own message when the child replies with an explicit cleanup failure, and the child is terminated.
  • Updated: the child signal-routing test now expects exit status 0 for SIGINT, SIGHUP, and SIGTERM.
  • Existing coverage for draining an active command before closing IPC, shutdown-receipt fencing, and pre-dispatch atomicity is unchanged and passes.

npm test in packages/code passes 382 of 386 locally. The 4 failures are storage tests that create fixtures under /mnt/c, a WSL DrvFS mount with mode 777, and are unrelated to this change.

Note

The commits on this branch are unsigned because GPG could not prompt for a passphrase in the environment it was authored from.

A native BYOM worker under systemd KillMode=control-group receives SIGTERM
at the same time as its forked SRT executor. The child ignores IPC once it
is shutting down, so the parent's close handshake is left pending until the
child exits, which rejects it with 'Native executor is unavailable'. That
rejection escaped the CLI finally block and turned an idle administrative
stop into exit status 1.

Treat the close handshake as best-effort: the executor is terminated in
finally regardless, and the active command has already drained, so a lost
or stalled reply carries no mutation risk. Also make the child report exit
status 0 when its own SRT teardown succeeded.

Closes #190
Only a lost, refused, or stalled close handshake is benign at shutdown.
A negative close reply from the executor is a real cleanup failure and
still rejects so pool shutdown can aggregate it.
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-13T08:04:34.855714Z 2ff14c8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 2ff14c8273

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit 8764d01 into main Sep 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native BYOM worker exits 1 during clean systemd shutdown

1 participant