Skip to content

fix(process): retain lifecycle events across in-place upgrades - #1033

Draft
wolfy-j wants to merge 1 commit into
mainfrom
fix/process-lifecycle-upgrade-20261031
Draft

wolfy-j wants to merge 1 commit into
mainfrom
fix/process-lifecycle-upgrade-20261031

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

An unread lifecycle event accepted before an in-place upgrade can disappear. In particular, a Jobs owner can lose a queued CANCEL and remain alive past its shutdown deadline, while a second CANCEL sent after the upgrade stops the same PID.

process.events() leaves RetainOnUpgrade false, so the mailbox transfer classifies its topic as belonging to a retiring producer and discards the queued tail. Mark lifecycle subscriptions as retained, using the existing mailbox-transfer behavior already used by process.inbox() and process.listen(). No scheduler, producer-retirement algorithm, subscription API, or release metadata changes.

The regression steps the real Lua process engine directly, without sleeps or a scheduler race. The old incarnation consumes one event and waits outside its lifecycle channel while CANCEL and EXIT remain unread. The replacement must receive those events in order exactly once, must not replay the consumed event, and must still accept a fresh lifecycle event. On the base commit the unread-event count is 0 instead of 2; with the fix the test passes.

Validation:

  • go test -race -short -count=1 -timeout 10m ./runtime/lua/modules/process ./runtime/lua/engine ./system/process ./system/topology ./service/supervisor
  • make lint LINT_PACKAGES='./runtime/lua/modules/process' — 0 issues.
  • Native standalone actor probe, with no Jobs or SQL: fails on released v0.3.56a; passes with the candidate.
  • Actual Jobs owner, paused by a test-only barrier at its existing upgrade call: passes on SQLite and PostgreSQL, both locally and on Towers. The PID and original 15-second EXIT bound are preserved; no retry loop or deadline increase. No LLM inference.
  • Relevant existing engine tests cover retirement of old producer frames as well as mailbox ordering and delivery.

The original Jobs shutdown test is timing-sensitive and can miss this ordering; the new Go regression forces it deterministically. This does not claim the Jobs performance-settings change introduced the defect. The full cross-platform CI suite still needs to run on this PR.

Draft for review by @skhaz. No Runtime release is requested here. KickSide #215 and #245 remain held until the upstream fix is reviewed, released, and qualified downstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants