Skip to content

Commit c376605

Browse files
authored
Merge pull request #214 from durable-workflow/fix/175-fenced-deregistration
Bound worker shutdown and fence deregistration recovery
2 parents e530e33 + 352cbcf commit c376605

19 files changed

Lines changed: 940 additions & 41 deletions

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
66

77
## [Unreleased]
88

9+
## [2.10.1] - Unreleased
10+
11+
### Fixed
12+
13+
- Keep worker deregistration inside the remaining shutdown deadline. When the
14+
runtime advertises generation-fenced receipts, retry matching transient or
15+
lost responses with the original registration token. Without advertised
16+
fencing, use one legacy attempt. Exhausted or uncertain shutdown keeps
17+
registration removal unconfirmed instead of reporting success.
18+
919
## [2.10.0] - 2026-10-23
1020

1121
### Added

‎docs/reference/worker.md‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,27 @@ worker-stop behavior. Neither uses this generic poll-failure backoff.
6565
before polling. After registration succeeds, shutdown stops every poller and
6666
waits for accepted work within the shared `shutdown_timeout` deadline before
6767
removing that worker-plane registration. Work still running at the deadline is
68-
cancelled. Calling `stop()` more than once is safe and sends at most one
69-
deregistration request. No deregistration is sent if registration failed.
68+
cancelled. Calling `stop()` more than once shares the same shutdown operation
69+
and result. No deregistration is sent if registration failed.
70+
71+
**Unreleased source behavior, prepared for SDK 2.10.1:** the final HTTP call and any recovery use the
72+
original remaining shutdown budget. If Server advertises
73+
`worker_deregistration_fencing`, Worker retains its registration token and uses
74+
the fenced receipt endpoint. Matching transient responses and transport losses
75+
can retry that same generation. A completed receipt can replay safely even if
76+
a replacement now owns the worker ID. Authentication, malformed or mismatched
77+
receipts, lost authority and expired receipts fail without retrying. Without
78+
advertised fencing, Worker keeps one attempt through the legacy worker-plane
79+
DELETE and does not replay an uncertain result. That legacy route has no
80+
generation fence. Keep worker IDs unique to a process, and do not reuse an ID
81+
until the prior worker has fully stopped.
82+
83+
If drain consumes the deadline, no late deregistration request is sent. A final
84+
HTTP timeout or exhausted recovery raises `RuntimeError` and leaves removal
85+
unconfirmed. The Server might have committed a request whose response was lost;
86+
an exception is not proof that the registration still exists. Inspect the
87+
runtime before taking an operator recovery action. These changes are not in a
88+
published package yet.
7089

7190
Cancelling an await cannot stop a running synchronous activity thread. Its
7291
physical slot remains occupied until the callback returns, and a late return

‎pyproject.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "durable-workflow"
7-
version = "2.10.0"
7+
version = "2.10.1"
88
description = "Python client and worker SDK for Durable Workflow Cloud and self-hosted Server"
99
readme = "README.md"
1010
requires-python = ">=3.10"
@@ -76,8 +76,8 @@ durable-workflow-replay-conformance = "durable_workflow.replay_conformance:main"
7676
durable-workflow-workflow-updates-conformance = "durable_workflow.workflow_updates_conformance:main"
7777

7878
[tool.durable-workflow]
79-
product-train = "2.10.0"
80-
registry-version = "2.10.0"
79+
product-train = "2.10.1"
80+
registry-version = "2.10.1"
8181
supported-server-versions = "2.8.0"
8282
worker-protocol-version = "1.20"
8383
control-plane-version = "2"

‎src/durable_workflow/_metric_routes.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
for suffix in ("", "/pass")),
1313
"/workers", "/workers/{worker_id}",
1414
"/worker/register", "/worker/heartbeat", "/worker/registrations/{worker_id}",
15+
"/worker/registrations/{worker_id}/deregister",
1516
"/worker/sessions", "/worker/sessions/{session_id}", "/worker/sessions/{session_id}/heartbeat",
1617
*(f"/worker/{kind}-tasks/poll" for kind in ("workflow", "activity", "query", "update-validation")),
1718
*(f"/worker/{kind}-tasks/{{task_id}}/{action}" for kind, actions in (

‎src/durable_workflow/client.py‎

Lines changed: 40 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1853,6 +1853,7 @@ async def _request(
18531853
json: Any = None,
18541854
timeout: float | None = None,
18551855
context: str = "",
1856+
retry: bool = True,
18561857
) -> Any:
18571858
if worker and discovery:
18581859
raise ValueError("a request cannot be both worker-plane and cluster discovery")
@@ -1920,7 +1921,7 @@ async def _do_request() -> httpx.Response:
19201921

19211922
try:
19221923
try:
1923-
resp = await self.retry_policy.execute(_do_request)
1924+
resp = await self.retry_policy.execute(_do_request) if retry else await _do_request()
19241925
except httpx.HTTPStatusError as exc:
19251926
status_code = str(exc.response.status_code)
19261927
outcome = "http_error"
@@ -5245,39 +5246,69 @@ async def close_worker_session(
52455246
json={"worker_id": worker_id, "reason": reason},
52465247
))
52475248

5248-
async def deregister_worker_registration(self, worker_id: str) -> dict[str, Any]:
5249+
async def deregister_worker_registration(
5250+
self, worker_id: str, *, registration_token: str | None = None, timeout: float | None = None,
5251+
expected_namespace: str | None = None,
5252+
) -> dict[str, Any]:
52495253
"""Remove this runtime's successful worker-plane registration.
52505254
52515255
Called by :class:`~durable_workflow.Worker` after its pollers and
52525256
in-flight tasks have drained. This is distinct from the operator-only
52535257
:meth:`deregister_worker` control-plane management operation. Returns
52545258
the worker deregistration envelope, including recovered workflow-task
5255-
count.
5259+
count. With a registration token, use the runtime's generation-fenced
5260+
receipt endpoint. Each call makes one attempt: Worker owns safe retries
5261+
and the remaining shutdown budget. An unfenced DELETE cannot safely be
5262+
repeated after an uncertain response because the worker ID may be reused.
5263+
``expected_namespace`` is a local registration-scope precondition, not
5264+
a namespace override.
52565265
"""
5266+
if expected_namespace is not None and self.namespace != expected_namespace:
5267+
raise RuntimeError("worker registration namespace changed; registration removal is unconfirmed")
5268+
if registration_token is not None and (
5269+
not isinstance(registration_token, str) or re.fullmatch(r"[a-f0-9]{32}", registration_token) is None
5270+
):
5271+
raise ValueError("registration_token must contain exactly 32 lowercase hexadecimal characters")
5272+
fenced = registration_token is not None
52575273
try:
52585274
data = await self._request(
5259-
"DELETE",
5260-
f"/worker/registrations/{quote(worker_id, safe='')}",
5275+
"POST" if fenced else "DELETE",
5276+
f"/worker/registrations/{quote(worker_id, safe='')}" + ("/deregister" if fenced else ""),
52615277
worker=True,
52625278
context=worker_id,
5279+
json={"registration_token": registration_token} if fenced else None,
5280+
timeout=timeout,
5281+
retry=False,
52635282
)
52645283
except ServerError as error:
5265-
if error.status == 404 and error.reason() == "worker_not_found":
5284+
if not fenced and error.status == 404 and error.reason() == "worker_not_found":
52665285
return {
52675286
"worker_id": worker_id,
52685287
"outcome": "already_deregistered",
52695288
"recovered_workflow_task_count": 0,
52705289
}
52715290
raise
5272-
if not isinstance(data, dict):
5291+
if not self._valid_worker_deregistration_response(data, worker_id, registration_token):
52735292
raise ServerError(
52745293
200,
52755294
{
52765295
"reason": "invalid_worker_deregistration_response",
5277-
"message": f"expected JSON object, got {type(data).__name__}",
5296+
"message": "expected a deregistration receipt for this worker registration",
52785297
},
52795298
)
5280-
return data
5299+
return cast(dict[str, Any], data)
5300+
5301+
@staticmethod
5302+
def _valid_worker_deregistration_response(data: Any, worker_id: str, token: str | None) -> bool:
5303+
if not isinstance(data, dict):
5304+
return False
5305+
count = data.get("recovered_workflow_task_count")
5306+
return (
5307+
data.get("worker_id") == worker_id
5308+
and data.get("outcome") == "deregistered"
5309+
and type(count) is int and count >= 0
5310+
and (token is None or data.get("registration_token") == token)
5311+
)
52815312

52825313
async def heartbeat_worker(
52835314
self,

‎src/durable_workflow/errors.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,21 @@ def reason(self) -> str | None:
219219
return self.body.get("reason")
220220
return None
221221

222+
def worker_deregistration_retry_delay(self, worker_id: str, registration_token: str) -> int | None:
223+
"""Return a delay only for an identity-preserving fenced transient."""
224+
body = self.body
225+
if (
226+
self.status != 503 or not isinstance(body, dict)
227+
or body.get("reason") not in ("backend_lock_pressure", "backend_unavailable")
228+
or body.get("operation") != "deregister_worker" or body.get("outcome") != "unknown"
229+
or body.get("worker_id") != worker_id or body.get("registration_token") != registration_token
230+
or body.get("retryable") is not True
231+
or type(body.get("retry_after_seconds")) is not int
232+
or not 0 < body["retry_after_seconds"] <= 60
233+
):
234+
return None
235+
return int(body["retry_after_seconds"])
236+
222237
def poll_capacity_backpressure_delay(self, task_kind: str, task_queue: str) -> int | None:
223238
"""Return the advertised wait only for a complete poll-capacity refusal."""
224239
body = self.body

‎src/durable_workflow/worker.py‎

Lines changed: 96 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@
3737
from types import FunctionType
3838
from typing import Annotated, Any, Concatenate, Literal, ParamSpec, TypeVar, Union, get_args, get_origin, get_type_hints
3939

40+
import httpx
41+
4042
from . import serializer
4143
from ._activity_lease import activity_lease_renewal_supported, current_activity_authority
4244
from ._activity_process import CallbackFailure, CallbackInvocation, CallbackProcessLost, SupervisedCallback
@@ -1169,6 +1171,10 @@ def __init__(
11691171
self._in_flight: set[asyncio.Task[Any]] = set()
11701172
self._poller_tasks: set[asyncio.Task[Any]] = set()
11711173
self._registered = False
1174+
self._deregistration_fenced = False
1175+
self._registration_token: str | None = None
1176+
self._registration_worker_id: str | None = None
1177+
self._registration_namespace: object = None
11721178
self._run_started = False
11731179
self._registration_done = asyncio.Event()
11741180
self._shutdown_task: asyncio.Task[None] | None = None
@@ -1295,6 +1301,18 @@ async def _register(self) -> None:
12951301
_validate_server_compatibility(info)
12961302
protocol = info.get("worker_protocol")
12971303
server_capabilities = protocol.get("server_capabilities") if isinstance(protocol, Mapping) else None
1304+
deregistration = (
1305+
server_capabilities.get("worker_deregistration_fencing")
1306+
if isinstance(server_capabilities, Mapping) else None
1307+
)
1308+
self._deregistration_fenced = isinstance(deregistration, Mapping) and deregistration.get("supported") is True
1309+
if isinstance(deregistration, Mapping) and self._deregistration_fenced and (
1310+
deregistration.get("schema") != "durable-workflow.v2.worker-deregistration.v1"
1311+
or deregistration.get("endpoint") != "/worker/registrations/{workerId}/deregister"
1312+
or type(deregistration.get("receipt_retention_seconds")) is not int
1313+
or deregistration["receipt_retention_seconds"] <= 0
1314+
):
1315+
raise RuntimeError("Server compatibility error: invalid worker_deregistration_fencing contract")
12981316
sticky_support = (
12991317
server_capabilities.get("sticky_execution") if isinstance(server_capabilities, Mapping) else None
13001318
)
@@ -1449,6 +1467,20 @@ async def _register(self) -> None:
14491467
task_slots=self._current_task_slots(),
14501468
process_metrics=self._current_process_metrics(),
14511469
)
1470+
if self._deregistration_fenced:
1471+
token = ack.get("registration_token") if isinstance(ack, dict) else None
1472+
if (
1473+
not isinstance(token, str) or len(token) != 32
1474+
or any(character not in "0123456789abcdef" for character in token)
1475+
or ack.get("worker_id") != self.worker_id or ack.get("registered") is not True
1476+
):
1477+
raise RuntimeError(
1478+
"Server returned an invalid registration_token acknowledgement; "
1479+
"registration may remain active and unfenced cleanup is unsafe"
1480+
)
1481+
self._registration_token = token
1482+
self._registration_worker_id = self.worker_id
1483+
self._registration_namespace = getattr(self.client, "namespace", None)
14521484
self._registered = True
14531485
# Adapt to the server-advertised cadence when present so a cluster
14541486
# can pin the worker fleet's heartbeat beat without each worker
@@ -4567,8 +4599,9 @@ def _track_worker_session_from_task(self, task: dict[str, Any]) -> None:
45674599
async def stop(self) -> None:
45684600
"""Stop polling, drain tasks, and remove the worker registration.
45694601
4570-
Repeated calls share one shutdown result. A failed deregistration is
4571-
therefore propagated to every caller without sending another DELETE.
4602+
Repeated calls share one shutdown result. Generation-fenced recovery
4603+
stays within the original shutdown deadline. Unfenced deregistration
4604+
makes one attempt and does not retry an uncertain outcome.
45724605
"""
45734606
self._stop.set()
45744607
if self._shutdown_task is None:
@@ -4664,8 +4697,9 @@ async def _shutdown(self) -> None:
46644697
log.warning("worker session %s close failed: %s", session.options.session_id, error)
46654698

46664699
if self._registered:
4667-
await self.client.deregister_worker_registration(self.worker_id)
4700+
await self._deregister_before_deadline(deadline)
46684701
self._registered = False
4702+
self._registration_token = None
46694703
log.info("worker %s deregistered", self.worker_id)
46704704
self._sticky_cache.clear()
46714705
payload_observers = getattr(self.client, "_workflow_payload_observers", None)
@@ -4676,3 +4710,62 @@ async def _shutdown(self) -> None:
46764710
@staticmethod
46774711
def _remaining_shutdown_time(deadline: float) -> float:
46784712
return max(0.0, deadline - asyncio.get_running_loop().time())
4713+
4714+
def _check_deregistration_identity(self) -> None:
4715+
if self._deregistration_fenced and self._registration_token is None:
4716+
raise RuntimeError("worker registration token is missing; registration removal is unconfirmed")
4717+
if self._registration_worker_id is not None and (
4718+
self.worker_id != self._registration_worker_id
4719+
or getattr(self.client, "namespace", None) != self._registration_namespace
4720+
):
4721+
raise RuntimeError("worker registration identity changed; registration removal is unconfirmed")
4722+
4723+
async def _deregistration_attempt(self, remaining: float) -> tuple[float | None, Exception] | None:
4724+
kwargs: dict[str, Any] = {}
4725+
if self._deregistration_fenced:
4726+
kwargs = {"registration_token": self._registration_token, "timeout": remaining}
4727+
if isinstance(self._registration_namespace, str):
4728+
kwargs["expected_namespace"] = self._registration_namespace
4729+
try:
4730+
await asyncio.wait_for(
4731+
self.client.deregister_worker_registration(self.worker_id, **kwargs),
4732+
timeout=remaining,
4733+
)
4734+
except asyncio.TimeoutError as error:
4735+
raise RuntimeError(
4736+
"worker shutdown timed out during deregistration; registration removal is unconfirmed"
4737+
) from error
4738+
except ServerError as error:
4739+
retry_delay = (
4740+
error.worker_deregistration_retry_delay(self.worker_id, self._registration_token)
4741+
if self._deregistration_fenced and self._registration_token is not None else None
4742+
)
4743+
if retry_delay is None:
4744+
raise
4745+
return float(retry_delay), error
4746+
except (httpx.NetworkError, httpx.TimeoutException, httpx.RemoteProtocolError) as error:
4747+
if not self._deregistration_fenced:
4748+
raise
4749+
return None, error
4750+
return None
4751+
4752+
async def _deregister_before_deadline(self, deadline: float) -> None:
4753+
attempt = 0
4754+
while True:
4755+
self._check_deregistration_identity()
4756+
remaining = self._remaining_shutdown_time(deadline)
4757+
if remaining <= 0:
4758+
raise RuntimeError("worker shutdown deadline exhausted; registration removal is unconfirmed")
4759+
retry = await self._deregistration_attempt(remaining)
4760+
if retry is None:
4761+
return
4762+
retry_delay, failure = retry
4763+
delay = min(1.0, 0.1 * 2 ** min(attempt, 4)) if retry_delay is None else retry_delay
4764+
if delay >= self._remaining_shutdown_time(deadline):
4765+
raise RuntimeError(
4766+
"worker shutdown cannot retry within its deadline; registration removal is unconfirmed"
4767+
) from failure
4768+
if attempt == 0:
4769+
log.warning("worker deregistration is unconfirmed; retrying the fenced request within its deadline")
4770+
attempt += 1
4771+
await asyncio.sleep(delay)

‎tests/deregistration_recovery.py‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
"""Shared workflow and committed-response fault for deregistration controls."""
2+
from __future__ import annotations
3+
4+
from typing import Any
5+
6+
import httpx
7+
8+
from durable_workflow import Worker, activity, workflow
9+
10+
11+
@activity.defn(name="tests.deregistration.echo")
12+
def echo(value: str) -> str:
13+
return value
14+
15+
16+
@workflow.defn(name="tests.deregistration.cold-recovery")
17+
class RecoveredWorkflow:
18+
def run(self, ctx: Any, value: str) -> Any:
19+
returned = yield ctx.schedule_activity("tests.deregistration.echo", [value], retry_policy={"max_attempts": 1})
20+
return {"value": returned, "run": ctx.run_id}
21+
22+
23+
class DiscardCompletedResponse(httpx.AsyncBaseTransport):
24+
def __init__(self, replacement: Worker) -> None:
25+
self.inner = httpx.AsyncHTTPTransport()
26+
self.replacement = replacement
27+
self.discarded = False
28+
self.deregistration_bodies: list[bytes] = []
29+
self.recovered_workflow_task_count: int | None = None
30+
31+
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
32+
response = await self.inner.handle_async_request(request)
33+
if request.method != "POST" or not request.url.path.endswith("/deregister"):
34+
return response
35+
self.deregistration_bodies.append(request.content)
36+
if self.discarded:
37+
return response
38+
# The real Server commits before only the caller's response is lost.
39+
await response.aread()
40+
assert response.status_code == 200
41+
receipt = response.json()
42+
assert receipt["outcome"] == "deregistered"
43+
self.recovered_workflow_task_count = receipt["recovered_workflow_task_count"]
44+
await response.aclose()
45+
await self.replacement._register()
46+
assert self.replacement._registration_token != receipt["registration_token"]
47+
self.discarded = True
48+
raise httpx.RemoteProtocolError("test discarded a committed response", request=request)
49+
50+
async def aclose(self) -> None:
51+
await self.inner.aclose()

0 commit comments

Comments
 (0)