Skip to content

Commit 65a5b28

Browse files
committed
Add bounded checkpointed batch example and commit-boundary controls
1 parent edcd90f commit 65a5b28

4 files changed

Lines changed: 361 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ for the complete cross-SDK contract.
9999
- [Python SDK guide](https://durable-workflow.com/docs/2.0/polyglot/python/)
100100
- [Complete SDK reference](docs/sdk-reference.md)
101101
- [Runnable examples](examples/)
102+
- [Checkpointed batch source example](examples/CHECKPOINTED_BATCH.md) (unreleased accessor; synthetic provider)
102103
- [Separate workflow and activity worker processes](docs/guides/separate-workers.md)
103104
- [Symmetric SDK playground](https://github.com/durable-workflow/sample-app#symmetric-sdk-playground)
104105

‎examples/CHECKPOINTED_BATCH.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Resume a bounded batch
2+
3+
This source example uses the unreleased acknowledged-checkpoint API from
4+
[SDK333](https://github.com/durable-workflow/sdk-python/pull/333). It requires a
5+
Server advertising `activity_heartbeat_resume` v1. Its source tests do not prove
6+
published-package compatibility, real process loss or Server persistence.
7+
8+
On Linux, from a source checkout with its development environment, point it at
9+
an isolated development Server. Supply a **new** directory for each invocation.
10+
The synthetic provider receipts remain there for inspection; existing
11+
directories are refused.
12+
13+
```bash
14+
export SERVER_URL=http://127.0.0.1:8080
15+
export WORKFLOW_TOKEN=your-development-token
16+
python -m examples.checkpointed_batch ./receipts-before before-commit
17+
python -m examples.checkpointed_batch ./receipts-commit after-commit
18+
python -m examples.checkpointed_batch ./receipts-checkpoint after-checkpoint
19+
```
20+
21+
The activity processes three items, with a maximum of sixteen bounded items.
22+
Its two attempts, one-second retry delay, ten-second heartbeat timeout,
23+
thirty-second start-to-close timeout and sixty-second schedule-to-close timeout
24+
stay fixed. Recovery does not extend those budgets. The example's worker wait
25+
is bounded separately.
26+
27+
| First-attempt failure | Replacement action |
28+
| --- | --- |
29+
| Before provider commit | Commit item zero, then acknowledge progress |
30+
| After commit, before acknowledgement | Reconcile the stable provider key, then acknowledge progress |
31+
| After acknowledgement | Resume at the received next-item cursor |
32+
| Acknowledgement response lost | The next claim may contain the checkpoint or none; reconcile any repeated provider operation |
33+
34+
`last_heartbeat_details(result_type=Progress)` reads the immutable received
35+
snapshot. Unsupported claims refuse instead of inventing empty state. A
36+
supported empty checkpoint starts from zero. The typed cursor binds the full
37+
business input and must be within the item range.
38+
39+
Provider receipts are separate durable state. A key uses the business ID and
40+
item index, never an attempt ID. Each receipt also binds the entire batch
41+
fingerprint, so appending, truncating or reordering input under a repeated
42+
business ID refuses even without a checkpoint. Revisited keys require the exact
43+
receipt. Partial or conflicting receipts are preserved and require
44+
reconciliation. File and directory entries are synchronized before commit or
45+
reconciliation returns. This is a private synthetic provider, not an actual
46+
payment or other external service.
47+
48+
The ordering is provider commit/reconcile, then `await ctx.heartbeat(...)`, then
49+
continue. A failed heartbeat stops the attempt; an uncertain response does not
50+
mean the checkpoint failed to commit. The SDK snapshot does not advance in
51+
place. A checkpoint does not atomically commit an external business effect.
52+
Use the real provider's idempotency, operation lookup and fencing contract,
53+
retain the business ID across restarts of that operation, and choose a new ID
54+
for a new business operation. The example does not promise exactly-once effects.
55+
56+
The source tests use synthetic files and acknowledgement callbacks without a
57+
Server. They cover all three failure boundaries, both lost-response outcomes,
58+
changed input without a checkpoint, malformed cursors and partial receipts:
59+
60+
```bash
61+
pytest -q tests/test_checkpointed_batch_example.py
62+
```

‎examples/checkpointed_batch.py‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
"""Bounded source example using the unreleased checkpoint API. See CHECKPOINTED_BATCH.md."""
2+
from __future__ import annotations
3+
4+
import argparse
5+
import asyncio
6+
import hashlib
7+
import json
8+
import os
9+
from collections.abc import Awaitable, Callable, Generator
10+
from dataclasses import asdict, dataclass
11+
from pathlib import Path
12+
from typing import Any
13+
from uuid import uuid4
14+
15+
from durable_workflow import ActivityRetryPolicy, Client, Worker, activity, workflow
16+
from durable_workflow.errors import ActivityHeartbeatResumeUnavailable, NonRetryableError
17+
18+
FAILURES = ("before-commit", "after-commit", "after-checkpoint")
19+
ACTIVITY = "examples.checkpointed-batch"
20+
21+
22+
@dataclass(frozen=True)
23+
class Batch:
24+
business_id: str
25+
items: list[str]
26+
failure: str
27+
28+
def snapshot(self) -> tuple[str, tuple[str, ...], str]:
29+
if not isinstance(self.items, list) or not 1 <= len(self.items) <= 16 or self.failure not in FAILURES:
30+
raise NonRetryableError("use a business ID and one to sixteen bounded items")
31+
try:
32+
valid = all(isinstance(value, str) and 1 <= len(value.encode()) <= 128
33+
for value in [self.business_id, *self.items])
34+
except UnicodeEncodeError:
35+
valid = False
36+
if not valid:
37+
raise NonRetryableError("use a business ID and one to sixteen bounded items")
38+
items = tuple(self.items)
39+
encoded = json.dumps([self.business_id, items], ensure_ascii=False, separators=(",", ":")).encode()
40+
return self.business_id, items, hashlib.sha256(encoded).hexdigest()
41+
42+
43+
@dataclass(frozen=True)
44+
class Progress:
45+
batch_sha256: str
46+
next_item: int
47+
48+
49+
@dataclass(frozen=True)
50+
class ReceiptStore:
51+
"""Synthetic durable provider state, independent of Server checkpoints."""
52+
53+
root: Path
54+
55+
@classmethod
56+
def create(cls, root: Path) -> ReceiptStore:
57+
root.mkdir(mode=0o700)
58+
return cls(root)
59+
60+
def sync_directory(self) -> None:
61+
descriptor = os.open(self.root, os.O_RDONLY | os.O_DIRECTORY)
62+
try:
63+
os.fsync(descriptor)
64+
finally:
65+
os.close(descriptor)
66+
67+
def commit_or_reconcile(self, business_id: str, fingerprint: str, index: int, item: str) -> bool:
68+
key = json.dumps([business_id, index], ensure_ascii=False, separators=(",", ":")).encode()
69+
path = self.root / (hashlib.sha256(key).hexdigest() + ".json")
70+
receipt = json.dumps({"key": [business_id, index], "batch_sha256": fingerprint, "item": item},
71+
ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
72+
try:
73+
with path.open("xb") as stream:
74+
stream.write(receipt)
75+
stream.flush()
76+
os.fsync(stream.fileno())
77+
created = True
78+
except FileExistsError:
79+
with path.open("rb") as stream:
80+
if stream.read(4097) != receipt:
81+
raise NonRetryableError(
82+
"provider receipt conflicts or is partial; reconcile before retrying",
83+
) from None
84+
os.fsync(stream.fileno())
85+
created = False
86+
self.sync_directory()
87+
return created
88+
89+
90+
async def process_batch(
91+
store: ReceiptStore, batch: Batch, received: Progress | None, attempt: int,
92+
acknowledge: Callable[[Progress], Awaitable[None]],
93+
) -> dict[str, Any]:
94+
business_id, items, fingerprint = batch.snapshot()
95+
if type(attempt) is not int or attempt < 1:
96+
raise NonRetryableError("the activity attempt must be positive")
97+
if received is not None and (
98+
received.batch_sha256 != fingerprint or type(received.next_item) is not int
99+
or not 0 <= received.next_item <= len(items)
100+
):
101+
raise NonRetryableError("checkpoint belongs to different input or is out of range")
102+
start = received.next_item if received is not None else 0
103+
def fail(point: str, index: int) -> None:
104+
if attempt == 1 and index == 0 and batch.failure == point:
105+
raise RuntimeError("planned first-attempt failure at " + point)
106+
107+
for index in range(start, len(items)):
108+
fail("before-commit", index)
109+
await asyncio.to_thread(store.commit_or_reconcile, business_id, fingerprint, index, items[index])
110+
fail("after-commit", index)
111+
await acknowledge(Progress(fingerprint, index + 1))
112+
fail("after-checkpoint", index)
113+
return {"business_id": business_id, "completed": len(items)}
114+
115+
116+
@workflow.defn(name=ACTIVITY)
117+
class BatchWorkflow:
118+
def run(self, ctx: workflow.WorkflowContext, batch: dict[str, Any]) -> Generator[Any, Any, Any]:
119+
return (yield ctx.schedule_activity(
120+
ACTIVITY, [batch], retry_policy=ActivityRetryPolicy(max_attempts=2, backoff_seconds=[1]),
121+
heartbeat_timeout=10, start_to_close_timeout=30, schedule_to_close_timeout=60,
122+
))
123+
124+
125+
async def run(root: Path, failure: str) -> None:
126+
store = await asyncio.to_thread(ReceiptStore.create, root)
127+
128+
@activity.defn(name=ACTIVITY)
129+
async def execute(body: dict[str, Any]) -> dict[str, Any]:
130+
ctx = activity.context()
131+
try:
132+
received = ctx.last_heartbeat_details(result_type=Progress)
133+
except ActivityHeartbeatResumeUnavailable:
134+
raise NonRetryableError("this example requires activity_heartbeat_resume v1") from None
135+
136+
async def acknowledge(progress: Progress) -> None:
137+
await ctx.heartbeat(asdict(progress))
138+
139+
return await process_batch(store, Batch(**body), received, ctx.info.attempt_number, acknowledge)
140+
141+
queue = "checkpointed-batch-" + uuid4().hex
142+
async with Client(os.environ.get("SERVER_URL", "http://127.0.0.1:8080"),
143+
token=os.environ.get("WORKFLOW_TOKEN"), namespace="default", timeout=10) as client:
144+
worker = Worker(client, task_queue=queue, workflows=[BatchWorkflow], activities=[execute])
145+
handle = await client.start_workflow(
146+
workflow_type=ACTIVITY, task_queue=queue, workflow_id=queue,
147+
input=[asdict(Batch("demo-" + uuid4().hex, ["first", "second", "third"], failure))],
148+
)
149+
await worker.run_until(workflow_id=handle.workflow_id, timeout=70)
150+
print(json.dumps(await handle.result(timeout=3)))
151+
152+
153+
def main() -> None:
154+
parser = argparse.ArgumentParser(description=__doc__)
155+
parser.add_argument("receipts", type=Path, help="a new private synthetic-provider directory")
156+
parser.add_argument("failure", choices=FAILURES)
157+
args = parser.parse_args()
158+
asyncio.run(run(args.receipts, args.failure))
159+
160+
161+
if __name__ == "__main__":
162+
main()
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
"""Synthetic provider/acknowledgement boundaries; no Server or worker invocation."""
2+
from __future__ import annotations
3+
4+
from dataclasses import replace
5+
from pathlib import Path
6+
from typing import Any
7+
8+
import pytest
9+
from examples.checkpointed_batch import FAILURES, Batch, Progress, ReceiptStore, process_batch
10+
11+
from durable_workflow.errors import NonRetryableError
12+
13+
14+
def batch(failure: str = "after-commit") -> Batch:
15+
return Batch("business-batch", ["first", "second", "third"], failure)
16+
17+
18+
def inventory(store: ReceiptStore) -> list[tuple[str, bytes]]:
19+
return sorted((path.name, path.read_bytes()) for path in store.root.iterdir())
20+
21+
22+
@pytest.mark.parametrize("point", FAILURES)
23+
async def test_commit_boundaries_resume_only_acknowledged_items(tmp_path: Path, point: str) -> None:
24+
store = ReceiptStore.create(tmp_path / "provider")
25+
request = batch(point)
26+
persisted = None
27+
28+
async def acknowledge(progress: Progress) -> None:
29+
nonlocal persisted
30+
persisted = progress
31+
32+
with pytest.raises(RuntimeError, match="planned first-attempt failure"):
33+
await process_batch(store, request, None, 1, acknowledge)
34+
assert (persisted.next_item if persisted else None) == (1 if point == "after-checkpoint" else None)
35+
assert len(inventory(store)) == (0 if point == "before-commit" else 1)
36+
resumed = []
37+
38+
async def acknowledge_replacement(progress: Progress) -> None:
39+
resumed.append(progress.next_item)
40+
41+
assert await process_batch(store, request, persisted, 2, acknowledge_replacement) == {
42+
"business_id": "business-batch", "completed": 3,
43+
}
44+
assert resumed == ([2, 3] if point == "after-checkpoint" else [1, 2, 3])
45+
before = inventory(store)
46+
business_id, items, fingerprint = request.snapshot()
47+
for index, item in enumerate(items):
48+
assert store.commit_or_reconcile(business_id, fingerprint, index, item) is False
49+
assert inventory(store) == before
50+
51+
52+
@pytest.mark.parametrize("committed", [False, True])
53+
async def test_lost_ack_response_handles_both_commit_outcomes(tmp_path: Path, committed: bool) -> None:
54+
store = ReceiptStore.create(tmp_path / "provider")
55+
request = batch("after-checkpoint")
56+
persisted = None
57+
58+
async def lost_response(progress: Progress) -> None:
59+
nonlocal persisted
60+
if committed:
61+
persisted = progress
62+
raise TimeoutError("synthetic acknowledgement response lost")
63+
64+
with pytest.raises(TimeoutError):
65+
await process_batch(store, request, None, 1, lost_response)
66+
assert len(inventory(store)) == 1
67+
next_items = []
68+
69+
async def acknowledge(progress: Progress) -> None:
70+
next_items.append(progress.next_item)
71+
72+
assert (await process_batch(store, request, persisted, 2, acknowledge))["completed"] == 3
73+
assert next_items == ([2, 3] if committed else [1, 2, 3])
74+
assert len(inventory(store)) == 3
75+
76+
77+
@pytest.mark.parametrize("items", [
78+
["first", "second", "third", "fourth"], ["first", "second"], ["second", "first", "third"],
79+
])
80+
async def test_changed_input_without_checkpoint_preserves_provider_state(tmp_path: Path, items: list[str]) -> None:
81+
store = ReceiptStore.create(tmp_path / "provider")
82+
83+
async def no_op(_progress: Progress) -> None:
84+
pass
85+
86+
request = batch()
87+
await process_batch(store, request, None, 2, no_op)
88+
before = inventory(store)
89+
90+
async def forbidden(_progress: Progress) -> None:
91+
raise AssertionError("conflicting business input was acknowledged")
92+
93+
with pytest.raises(NonRetryableError, match="provider receipt conflicts"):
94+
await process_batch(store, replace(request, items=items), None, 2, forbidden)
95+
assert inventory(store) == before
96+
97+
98+
@pytest.mark.parametrize("cursor", [Progress("foreign", 1), Progress("same", -1),
99+
Progress("same", 4), Progress("same", True)])
100+
async def test_bad_checkpoint_refuses_before_provider_access(tmp_path: Path, cursor: Progress) -> None:
101+
store = ReceiptStore.create(tmp_path / "provider")
102+
request = batch()
103+
if cursor.batch_sha256 == "same":
104+
cursor = replace(cursor, batch_sha256=request.snapshot()[2])
105+
106+
async def forbidden(_progress: Progress) -> None:
107+
raise AssertionError("invalid checkpoint was acknowledged")
108+
109+
with pytest.raises(NonRetryableError, match="checkpoint belongs"):
110+
await process_batch(store, request, cursor, 2, forbidden)
111+
assert inventory(store) == []
112+
113+
114+
@pytest.mark.parametrize("items", [[], ["item"] * 17, [""], ["x" * 129], ["\ud800"], [False], "not-a-list"])
115+
async def test_invalid_batch_refuses_before_provider_access(tmp_path: Path, items: Any) -> None:
116+
store = ReceiptStore.create(tmp_path / "provider")
117+
118+
async def forbidden(_progress: Progress) -> None:
119+
raise AssertionError("invalid batch was acknowledged")
120+
121+
with pytest.raises(NonRetryableError):
122+
await process_batch(store, Batch("business", items, "after-commit"), None, 2, forbidden)
123+
assert inventory(store) == []
124+
125+
126+
def test_existing_directory_and_partial_receipt_are_preserved(tmp_path: Path) -> None:
127+
store = ReceiptStore.create(tmp_path / "provider")
128+
with pytest.raises(FileExistsError):
129+
ReceiptStore.create(store.root)
130+
business_id, items, fingerprint = batch().snapshot()
131+
assert store.commit_or_reconcile(business_id, fingerprint, 0, items[0])
132+
path = next(store.root.iterdir())
133+
path.write_bytes(b"partial")
134+
with pytest.raises(NonRetryableError, match="partial"):
135+
store.commit_or_reconcile(business_id, fingerprint, 0, items[0])
136+
assert path.read_bytes() == b"partial"

0 commit comments

Comments
 (0)