fix(celery): dead-letter OOM-loop tasks by keying worker-loss cap on stable Mongo task id

[ocervell]

Problem (RC#6 — dead-letter broker-redelivery OOM loops)

A task that OOM-kills its worker (or is node-evicted) has its message requeued by the broker under task_acks_late + task_reject_on_worker_lost, and re-run. Secator already caps this in run_command via bump_worker_loss_count + abandon_task, gated on task_max_retries — but the counter was keyed on self.request.id, assuming the Celery id is stable across redeliveries.

For the fan-in "zombies" seen in prod (52,337 and 303,499-result fan-ins), that assumption broke: they redelivered via pure broker redelivery (redelivered=true, retries=0) presenting a fresh/rotating Celery id each cycle. So bump_worker_loss_count(self.request.id) reset to 1 on every delivery, the cap never tripped, and one worker pod was OOM-killed per cycle indefinitely — even after the parent scan had FAILED.

Fix

Key the counter on the stable Mongo task id instead of the rotating Celery id:

• New worker_loss_key(context, celery_id) → prefers context['task_chunk_id'] / context['task_id'] (minted at on_build, baked into the child signature's serialized context, so it rides inside the message body and is identical on every redelivery). Falls back to the Celery id for bare top-level tasks that never went through on_build (those keep a stable Celery id under reject_on_worker_lost, so prior behaviour is preserved).
• The count now accumulates on the logical task regardless of Celery-id rotation, so abandon_task dead-letters it after N deliveries — forwarding results (forward_results + chain_results) so the surrounding chord/workflow still completes instead of hanging.
• The bump already runs first thing in run_command, before forward_results/task setup, so it counts even when the OOM happens during setup.

No config or behavioural change when the cap is disabled (task_max_retries=-1) or when a task already keeps a stable Celery id.

Native-Celery note (for the reader)

Celery has no built-in delivery-count / dead-letter cap for the Redis broker. task_reject_on_worker_lost only re-queues on worker loss (it does not count or cap); Task.retry() is a different, in-task path (retries header) not involved here (retries=0); AMQP x-death / dead-letter exchanges are RabbitMQ-only and unavailable on the Redis broker + result-backend counting secator uses. So the counting + cap must live in application code — this PR keeps it there and only corrects the key. See the postmortem RC#6 research summary for detail.

Test

tests/unit/test_celery.py::TestWorkerLossRetryCap:

• test_redeliveries_with_rotating_celery_ids_abandon_after_cap — same logical task redelivered 7× with a different Celery id each time abandons on the 5th delivery (max_retries=3), and asserts the old per-Celery-id keying never accumulates past 1.
• test_worker_loss_key_prefers_mongo_id_over_rotating_celery_id — precedence: chunk id → task id → Celery id fallback.

secator test unit --test test_celery → 25 passed, 2 skipped. secator test lint clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MtTyzcUmPYxM5nfp7MnMVd

Summary by CodeRabbit

• Bug Fixes

  • Improved task redelivery tracking by consistently using a stable task identity.
  • Prevented repeated worker-loss redeliveries from resetting retry counts when delivery identifiers change.
  • Ensured tasks are abandoned at the expected retry limit, improving worker-loss recovery and preventing excessive duplicate processing.

• Tests

  • Added coverage for stable task identification and multi-redelivery abandonment behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0b0d9773-e3e7-4d42-b0e4-3b32bf74652a

📥 Commits

Reviewing files that changed from the base of the PR and between 8fcafa6 and 9da8cf9.

📒 Files selected for processing (2)

• secator/celery.py
• tests/unit/test_celery.py

---

Walkthrough

Worker-loss retry counting now uses stable task or chunk identifiers from task context, falling back to the Celery request ID. Redelivery tests cover identifier priority and abandonment after the configured cap despite rotating Celery IDs.

Changes

Worker-loss retry identity

Layer / File(s)	Summary
Stable key resolution secator/celery.py, tests/unit/test_celery.py	Adds logical task-key selection using task_chunk_id, then task_id, with Celery ID fallback; documents the updated counter parameter semantics and tests priority.
Redelivery cap integration secator/celery.py, tests/unit/test_celery.py	Routes worker-loss counting through the stable key and verifies abandonment across rotating Celery request IDs.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• freelabz/secator#1284: Changes the worker-loss counter keying in the same Celery redelivery logic.

Suggested labels: feature:worker-reliability

Poem

A rabbit watched the workers hop,
And kept one key that wouldn’t swap.
Through IDs that twirled in broker air,
The retry count stayed true and fair.
“Enough,” it said, “the cap is there!”

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main fix: dead-lettering Celery worker-loss loops by using a stable Mongo task id.
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/deadletter-stable-id

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}