fix: Exit Cleanly When Native Executor Shuts Down Concurrently - #191
Merged
Merged
Conversation
A native BYOM worker under systemd KillMode=control-group receives SIGTERM at the same time as its forked SRT executor. The child ignores IPC once it is shutting down, so the parent's close handshake is left pending until the child exits, which rejects it with 'Native executor is unavailable'. That rejection escaped the CLI finally block and turned an idle administrative stop into exit status 1. Treat the close handshake as best-effort: the executor is terminated in finally regardless, and the active command has already drained, so a lost or stalled reply carries no mutation risk. Also make the child report exit status 0 when its own SRT teardown succeeded. Closes #190
Only a lost, refused, or stalled close handshake is benign at shutdown. A negative close reply from the executor is a real cleanup failure and still rejects so pool shutdown can aggregate it.
Collaborator
Author
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A native BYOM worker running under systemd with
KillMode=control-groupexited with status 1 on an ordinarysystemctl stoporrestart, even while idle, and the journal recordedNative executor is unavailable. The worker restarted and reconnected fine, so every deploy looked like a service failure without being one. Closes #190.What raced. systemd delivers SIGTERM to the parent CLI and the forked SRT executor at the same moment. The child marks itself shutting down, begins closing its SRT sandbox, and ignores further IPC. The parent aborts the worker loop and reaches
nativeCommandSandbox.close()in the CLIfinallyalmost immediately. If the child is still connected at that point,stop()sends thecloseRPC, the child drops it, and when the child finally exits thelosthandler rejects the pending RPC withNative executor is unavailable. That rejection propagated throughclose(), the CLIfinally, andmain().catch()into exit status 1. The same sequence applies to Ctrl-C and any supervisor that signals the whole process group.closehandshake inNativeProcessWorkspaceCommandSandbox.stop(). The child is terminated infinallywhether or not it replies, andstop()has already drained any active command, so those cases carry no mutation ambiguity. A negativeclosereply from the executor, such as a failed SRT reset or scratch cleanup, is a real failure and still rejects soNativeWorkspaceCommandPool.close()can aggregate it. Quarantine decisions live on the execute path and are unchanged.Tests
close()resolves when the child exits during the close handshake without replying. This fails onmainwith the sameunavailablerejection seen in the issue.close()skips the handshake entirely once the child is already lost, and laterexecute()calls remain fenced.close()still rejects with the executor's own message when the child replies with an explicit cleanup failure, and the child is terminated.npm testinpackages/codepasses 382 of 386 locally. The 4 failures are storage tests that create fixtures under/mnt/c, a WSL DrvFS mount with mode 777, and are unrelated to this change.Note
The commits on this branch are unsigned because GPG could not prompt for a passphrase in the environment it was authored from.