Skip to content

Stop the shutdown handler from taking the shutdown event's lock - #1291

Merged
SimonHeybrock merged 1 commit into
mainfrom
service-shutdown-flake
Sep 7, 2026
Merged

Stop the shutdown handler from taking the shutdown event's lock#1291
SimonHeybrock merged 1 commit into
mainfrom
service-shutdown-flake

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

test_signal_during_startup_shuts_down_cleanly fails intermittently: the service ignores the SIGTERM, the test's 30 s wait expires, and the process has to be killed (returncode: -9).

The deadlock

The SIGTERM/SIGINT handler set _shutdown_requested before raising SystemExit. run_forever waits on that same event, and Event.wait holds the event's condition lock across a stretch of bytecodes on the way in and out. A signal delivered in that stretch runs the handler on a main thread that already owns the lock, and Event.set blocks on it forever -- the lock is not reentrant. Every later signal deadlocks in the same place, so only SIGKILL clears it.

Confirmed with a faulthandler dump of a hung service:

File "threading.py", line 300 in __enter__
File "threading.py", line 623 in set
File "src/ess/livedata/core/service.py", line 67 in _handle_shutdown

The handler's own docstring already stated the rule it was breaking: it must not take a lock the interrupted frame may hold. 10c0f07 established that rule for logging and introduced this instance of it for the event. Setting the event was redundant anyway -- SystemExit unwinds the main thread out of the wait on its own, and stop sets the event on the way out.

Why it started showing up

Nothing in #1256 or #1257 touches this code; what they changed is how tests are scheduled against each other. The window is sub-microsecond on an idle machine -- 600 stress attempts locally, idle and under CPU contention, produced no hang -- but a main thread preempted inside it stays there for as long as it is descheduled, which is what a contended runner does to it.

Worth stating: I could not reproduce the natural race locally. The mechanism is proven reachable and matches the symptom exactly, but I cannot strictly rule out a second cause behind the same failure. The alternatives I walked -- the worker-thread join, the logging lock, an unstarted thread at interpreter shutdown -- cannot hang for 30 s here.

The new test delivers the signal inside the window on purpose, so the deadlock is deterministic rather than a rare failure.

Test plan

  • New test reproduces the deadlock as a 30 s TimeoutExpired before the fix, passes after
  • Full suite with slow tests: 4983 passed, 82 skipped, 8 xfailed

The SIGTERM/SIGINT handler set `_shutdown_requested` before raising
SystemExit. `run_forever` waits on that same event, and `Event.wait` holds
the event's condition lock across a stretch of bytecodes on the way in and
out. A signal delivered in that stretch runs the handler on a main thread
that already owns the lock, and `Event.set` blocks on it forever -- the
lock is not reentrant. Further signals deadlock in the same place, so only
SIGKILL clears it.

That is the flake in test_signal_during_startup_shuts_down_cleanly: the
test aims a signal at the startup window, the main thread is a few
microseconds past it and inside `Event.wait`, and the service hangs until
the test's 30 s wait expires and kills it (exit -9). The window is
sub-microsecond on an idle box -- 600 attempts locally produced none -- but
a preempted main thread widens it to however long it stays descheduled,
which is why it only shows up on contended CI runners.

The handler now records the signal and raises SystemExit, nothing else.
Setting the event bought nothing: SystemExit already unwinds the main
thread out of the wait, and `stop` sets the event on the way out.

The new test reaches into the event's condition to deliver the signal
inside the window on purpose, so the deadlock is deterministic rather than
a rare CI failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@SimonHeybrock
SimonHeybrock merged commit adbe37e into main Sep 7, 2026
19 checks passed
@SimonHeybrock
SimonHeybrock deleted the service-shutdown-flake branch September 7, 2026 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant