Stop the shutdown handler from taking the shutdown event's lock - #1291
Merged
Conversation
The SIGTERM/SIGINT handler set `_shutdown_requested` before raising SystemExit. `run_forever` waits on that same event, and `Event.wait` holds the event's condition lock across a stretch of bytecodes on the way in and out. A signal delivered in that stretch runs the handler on a main thread that already owns the lock, and `Event.set` blocks on it forever -- the lock is not reentrant. Further signals deadlock in the same place, so only SIGKILL clears it. That is the flake in test_signal_during_startup_shuts_down_cleanly: the test aims a signal at the startup window, the main thread is a few microseconds past it and inside `Event.wait`, and the service hangs until the test's 30 s wait expires and kills it (exit -9). The window is sub-microsecond on an idle box -- 600 attempts locally produced none -- but a preempted main thread widens it to however long it stays descheduled, which is why it only shows up on contended CI runners. The handler now records the signal and raises SystemExit, nothing else. Setting the event bought nothing: SystemExit already unwinds the main thread out of the wait, and `stop` sets the event on the way out. The new test reaches into the event's condition to deliver the signal inside the window on purpose, so the deadlock is deterministic rather than a rare CI failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_signal_during_startup_shuts_down_cleanlyfails intermittently: the service ignores the SIGTERM, the test's 30 s wait expires, and the process has to be killed (returncode: -9).The deadlock
The SIGTERM/SIGINT handler set
_shutdown_requestedbefore raisingSystemExit.run_foreverwaits on that same event, andEvent.waitholds the event's condition lock across a stretch of bytecodes on the way in and out. A signal delivered in that stretch runs the handler on a main thread that already owns the lock, andEvent.setblocks on it forever -- the lock is not reentrant. Every later signal deadlocks in the same place, so only SIGKILL clears it.Confirmed with a faulthandler dump of a hung service:
The handler's own docstring already stated the rule it was breaking: it must not take a lock the interrupted frame may hold. 10c0f07 established that rule for logging and introduced this instance of it for the event. Setting the event was redundant anyway --
SystemExitunwinds the main thread out of the wait on its own, andstopsets the event on the way out.Why it started showing up
Nothing in #1256 or #1257 touches this code; what they changed is how tests are scheduled against each other. The window is sub-microsecond on an idle machine -- 600 stress attempts locally, idle and under CPU contention, produced no hang -- but a main thread preempted inside it stays there for as long as it is descheduled, which is what a contended runner does to it.
Worth stating: I could not reproduce the natural race locally. The mechanism is proven reachable and matches the symptom exactly, but I cannot strictly rule out a second cause behind the same failure. The alternatives I walked -- the worker-thread join, the logging lock, an unstarted thread at interpreter shutdown -- cannot hang for 30 s here.
The new test delivers the signal inside the window on purpose, so the deadlock is deterministic rather than a rare failure.
Test plan
TimeoutExpiredbefore the fix, passes after