Skip to content

Commit bf96d97

Browse files
committed
Complete deterministic parallel and saga authoring parity in Rust and Python
1 parent dfdc815 commit bf96d97

12 files changed

Lines changed: 867 additions & 39 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,14 @@ Earlier SDK versions remain historical releases and are not alternate supported
1414
2.0 baselines; no prerelease compatibility shim is provided.
1515

1616
### Added
17+
- List-yield parallel composition now accepts nested activity, child-workflow,
18+
and timer groups, schedules every durable leaf through the existing command
19+
protocol, restores the nested input shape on replay, and keeps stable full
20+
group paths across restart and duplicate terminal delivery.
21+
- `WorkflowContext.saga()` provides deterministic reverse-order activity
22+
compensation for failures and cooperative cancellation.
23+
`SagaCompensationFailed` preserves both the initiating and compensation
24+
failures as structured diagnostics.
1725
- Clients and workers now externalize large Avro payloads through the
1826
authenticated namespace runtime using only the runtime URL, namespace, and
1927
role credential. Opaque references are resolved before decode with bounded

‎README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,54 @@ receipt = yield ctx.start_child_workflow(
186186
)
187187
```
188188

189+
## Deterministic parallel groups
190+
191+
Yield a list to schedule one durable parallel barrier. Lists can nest and mix
192+
activities, child workflows, and timers. The worker flattens only the Server
193+
commands, records a stable full `parallel_group_path` on every leaf, and
194+
returns results in the original nested input shape regardless of terminal
195+
delivery order:
196+
197+
```python
198+
results = yield [
199+
ctx.schedule_activity("load-profile", [customer_id]),
200+
[
201+
ctx.start_child_workflow("quote-shipping", [customer_id]),
202+
ctx.start_timer(5),
203+
],
204+
]
205+
profile, (shipping, _) = results
206+
```
207+
208+
One failed activity or child is thrown at the list-yield point by durable input
209+
position. Already recorded sibling completions remain replayable; late and
210+
exact duplicate terminal deliveries do not change the selected result.
211+
212+
## Saga compensation
213+
214+
`ctx.saga()` registers ordinary activity commands as compensations and runs
215+
them sequentially in reverse registration order after failure or cooperative
216+
cancellation:
217+
218+
```python
219+
def forward(saga):
220+
flight = yield ctx.schedule_activity("trip.reserve-flight", [])
221+
saga.add_compensation("trip.cancel-flight", [flight])
222+
223+
hotel = yield ctx.schedule_activity("trip.reserve-hotel", [])
224+
saga.add_compensation("trip.cancel-hotel", [hotel])
225+
226+
ctx.throw_if_cancellation_requested()
227+
yield ctx.schedule_activity("trip.charge", [])
228+
return {"status": "booked"}
229+
230+
return (yield from ctx.saga().run(forward))
231+
```
232+
233+
Compensation stops at its first failure. `SagaCompensationFailed` retains the
234+
initiating failure, compensation failure, activity type, and deterministic
235+
registration order as structured fields.
236+
189237
## Nexus service calls
190238

191239
Workflow code can call a registered Nexus service operation through

‎docs/reference/workflow.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,14 @@ operation durably. The Python worker records the accepted response or typed
1010
service failure as a side-effect marker, then replay resumes with
1111
`NexusOperationResult` or raises `NexusOperationFailed` at the yield point.
1212

13+
Yield a list for deterministic parallel composition. Lists may nest and may
14+
mix activity, child-workflow, and timer commands; the resolved value has the
15+
same nested shape and input order. Every leaf emits the shared
16+
`parallel_group_*` metadata and uses its ordinary Server command.
17+
18+
Use `ctx.saga().run(forward)` for sequential reverse-order compensation.
19+
Register each compensation only after its forward activity completes. The
20+
helper compensates on failure or cooperative cancellation and raises
21+
`SagaCompensationFailed` if compensation itself fails.
22+
1323
::: durable_workflow.workflow

‎examples/parallel_saga.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
"""Workflow-only examples for deterministic parallel groups and sagas."""
2+
3+
from durable_workflow import ChildWorkflowFailed, workflow
4+
5+
6+
@workflow.defn(name="parallel-trip-quote")
7+
class ParallelTripQuote:
8+
def run(self, ctx): # type: ignore[no-untyped-def]
9+
try:
10+
return (
11+
yield [
12+
ctx.schedule_activity("trip.quote-flight", []),
13+
[
14+
ctx.start_child_workflow("trip.quote-hotel", []),
15+
ctx.start_timer(1),
16+
],
17+
]
18+
)
19+
except ChildWorkflowFailed as failure:
20+
return {"failed_child": failure.child_workflow_type, "message": str(failure)}
21+
22+
23+
@workflow.defn(name="trip-booking-saga")
24+
class TripBookingSaga:
25+
def run(self, ctx): # type: ignore[no-untyped-def]
26+
def forward(saga): # type: ignore[no-untyped-def]
27+
flight = yield ctx.schedule_activity("trip.reserve-flight", [])
28+
saga.add_compensation("trip.cancel-flight", [flight])
29+
30+
hotel = yield ctx.schedule_activity("trip.reserve-hotel", [])
31+
saga.add_compensation("trip.cancel-hotel", [hotel])
32+
33+
ctx.throw_if_cancellation_requested()
34+
yield ctx.schedule_activity("trip.charge", [])
35+
return {"flight": flight, "hotel": hotel}
36+
37+
# A worker restart or completed-history replay reconstructs every
38+
# registration. Compensation failure is surfaced as the exported
39+
# SagaCompensationFailed type with both failures retained.
40+
return (yield from ctx.saga().run(forward))

‎pyproject.toml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,15 @@ workflow-streams-operations = "list,describe,subscribe,append,close,error"
7878
workflow-streams-authoring-command = "record_side_effect.workflow_stream"
7979
workflow-streams-delivery = "at-least-once"
8080
workflow-streams-external-payload-references = "read-write"
81+
deterministic-parallel = true
82+
parallel-authoring = "nested-list-yield"
83+
parallel-members = "activity,child,timer,mixed"
84+
parallel-result-order = "input-position"
85+
parallel-group-metadata = "parallel_group_id,parallel_group_kind,parallel_group_base_sequence,parallel_group_size,parallel_group_index,parallel_group_path"
86+
saga-compensation = true
87+
saga-compensation-order = "reverse-registration-sequential"
88+
saga-compensation-failure-policy = "stop-first-preserve-initiating-and-compensation"
89+
saga-cancellation = "cooperative"
8190

8291
[tool.setuptools.packages.find]
8392
where = ["src"]

‎src/durable_workflow/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@
7979
QueryFailed,
8080
RuntimeCapabilityUnsupported,
8181
RuntimeDiscoveryUnavailable,
82+
SagaCompensationFailed,
8283
ScheduleAlreadyExists,
8384
ScheduleListError,
8485
ScheduleNotFound,
@@ -214,6 +215,7 @@
214215
NexusServiceCall,
215216
Replayer,
216217
ReplayOutcome,
218+
Saga,
217219
StartChildWorkflow,
218220
replay,
219221
)
@@ -247,6 +249,8 @@
247249
"NonRetryableError",
248250
"ReplayOutcome",
249251
"Replayer",
252+
"Saga",
253+
"SagaCompensationFailed",
250254
"ScheduleAction",
251255
"ScheduleAlreadyExists",
252256
"ScheduleBackfillResult",

‎src/durable_workflow/errors.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -541,6 +541,32 @@ def __init__(self, message: str = "workflow was terminated") -> None:
541541
super().__init__(message)
542542

543543

544+
class SagaCompensationFailed(DurableWorkflowError):
545+
"""A saga compensation failed after an earlier workflow failure.
546+
547+
Both failures remain available as typed attributes so applications and
548+
failure serializers do not have to recover the initiating cause from
549+
formatted text.
550+
"""
551+
552+
def __init__(
553+
self,
554+
initiating_failure: BaseException,
555+
compensation_failure: BaseException,
556+
*,
557+
compensation_activity_type: str,
558+
compensation_registration_order: int,
559+
) -> None:
560+
super().__init__(
561+
f"saga compensation {compensation_activity_type!r} failed after "
562+
f"{type(initiating_failure).__name__}: {initiating_failure}"
563+
)
564+
self.initiating_failure = initiating_failure
565+
self.compensation_failure = compensation_failure
566+
self.compensation_activity_type = compensation_activity_type
567+
self.compensation_registration_order = compensation_registration_order
568+
569+
544570
class WorkflowCancelled(BaseException):
545571
"""A workflow was cancelled and finished in the ``cancelled`` state.
546572

‎src/durable_workflow/worker.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1229,6 +1229,7 @@ async def _resolve_workflow_nexus_commands(
12291229
payload_codec=payload_codec,
12301230
external_storage=self.external_storage,
12311231
external_storage_cache=self.external_storage_cache,
1232+
cancel_requested=bool(task.get("cancel_requested", False)),
12321233
)
12331234
commands = list(outcome.commands)
12341235

@@ -1383,6 +1384,7 @@ async def _run_workflow_task_core(self, task: dict[str, Any]) -> list[dict[str,
13831384
payload_codec=codec,
13841385
external_storage=self.external_storage,
13851386
external_storage_cache=self.external_storage_cache,
1387+
cancel_requested=bool(task.get("cancel_requested", False)),
13861388
)
13871389
except AvroNotInstalledError as e:
13881390
log.exception("replay failed: Avro dependency unavailable")

0 commit comments

Comments
 (0)