feat(celery): finalize in-flight task on worker eviction so the chord proceeds

[ocervell]

Re-extracted from the closed #1203 — like the cap PR (#1284), this generic worker-reliability fix was folded into the ai-resiliency branch (#1241, AI-task hardening, base canary) and its standalone PR closed. It's general infra (applies to every runner, not AI), so it belongs on main independently. Cherry-picked clean onto current main (CI-retrigger noise commits dropped).

What it does

On a graceful worker shutdown (SIGTERM — k8s pod eviction / node drain / rolling update), worker_shutting_down sets a flag and the running task's monitor thread stops the task early and finalizes it — so the chord's mark_runner_completed still fires and the workflow proceeds, instead of the task hanging until the broker visibility timeout (hours on the long pool). Works on any pool (monitor-thread based). Per-task flag is cleared at start so a stale flag can't finalize a fresh task.

Validation

Exercised end-to-end earlier (SIGTERM → task stopped ~9s vs ~60s, chord proceeds, 5/5). tests/unit/test_eviction.py: 3 passed; test_celery.py: 17 passed on current main; flake8 clean.

Note

feature:worker-reliability. Companion: #1284 (worker-loss cap). #1202 (on_build identity) is already on main (in the pending 0.40.0).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added graceful shutdown handling for long-running workers so running jobs can detect eviction and stop sooner.
  • Improved monitoring responsiveness during shutdown checks to reduce the chance of stale work continuing after termination begins.

• Bug Fixes

  • Cleared stale shutdown state at startup to prevent old shutdown signals from affecting new runs.
  • Added safeguards so interrupted tasks emit a warning and terminate cleanly instead of hanging.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a803e7d4-6d4c-4c54-86bf-47aa9685232b

📥 Commits

Reviewing files that changed from the base of the PR and between 0203f02 and d10c332.

📒 Files selected for processing (3)

• secator/celery_signals.py
• secator/runners/command.py
• tests/unit/test_eviction.py

---

Walkthrough

Adds a file-based shutdown flag mechanism triggered by Celery's worker_shutting_down signal, enabling the command monitor thread to detect worker eviction, terminate running processes gracefully with partial results, and reduces monitor polling interval. Includes new unit tests covering the flag lifecycle and monitor behavior.

Changes

Worker Eviction Self-Finalization

Layer / File(s)	Summary
Shutdown flag helpers and signal wiring secator/celery_signals.py	Adds SHUTDOWN_FLAG constant and is_worker_shutting_down, clear_shutdown_flag, worker_shutting_down_handler functions; wires handler to Celery's worker_shutting_down signal and clears stale flags in setup_handlers().
Monitor thread shutdown detection and pacing secator/runners/command.py	Adds MONITOR_POLL_SECONDS constant; _monitor_process clears stale flags at start, checks is_worker_shutting_down() to stop the process with SIGTERM and enqueue a warning, and reduces the wait interval via min(CONFIG.runners.stat_update_frequency, MONITOR_POLL_SECONDS).
Unit tests for shutdown flag and monitor behavior tests/unit/test_eviction.py	Adds TestEvictionSelfFinalize verifying flag set/clear lifecycle, monitor stopping a process on flag detection with eviction warning enqueued, and monitor clearing stale flags at startup.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• freelabz/secator#727: Introduces the Command monitor thread that this PR extends with shutdown-flag checks inside _monitor_process.

Poem

A signal comes, the pod must go,
but tasks don't vanish—oh no, no!
A flag is set, the monitor sees,
it stops the process with graceful ease. 🐇
Partial results still find their way,
the chord proceeds—hooray, hooray!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: handling Celery worker eviction to finalize in-flight tasks.
Linked Issues check	✅ Passed	The PR implements the requested shutdown flag, monitor polling cap, stale-flag clearing, and tests for eviction-driven early completion.
Out of Scope Changes check	✅ Passed	The changes stay focused on Celery eviction handling and related tests, with no clear unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/worker-eviction-finalize-main

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ocervell]

Parked for later (owner decision). Concern: not enough confidence in the in-flight runner's behavior on worker shutdown — what happens to the running Python runner / external tool process, partial results, and how finalize-early interacts with redelivery. Too many edge cases to merge without more validation.

Safe to park: this only helps the graceful SIGTERM (k8s eviction / drain / rolling update) path — a hard worker death (SIGKILL/OOM) is handled by acks_late redelivery + the worker-loss cap (#1284) and the broker visibility-timeout fallback. So the platform works correctly without this; it just finalizes slower on graceful eviction. Revisit when we can validate the runner/partial-results semantics end-to-end.