Skip to content

Commit dadd05b

Browse files
committed
Adopt a fixed typed Avro Value protocol before stable 2.0
1 parent e8f21a0 commit dadd05b

17 files changed

Lines changed: 730 additions & 230 deletions

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ jobs:
2222
- run: pip install -e '.[dev]'
2323
- run: ruff check src/ tests/
2424
- run: mypy src/durable_workflow/
25+
- name: Enforce optimized Avro Value regression budget
26+
run: python benchmarks/avro_value.py --enforce
2527

2628
test:
2729
runs-on: ubuntu-latest
@@ -101,7 +103,7 @@ jobs:
101103
- name: Start server stack
102104
working-directory: sdk-python
103105
run: |
104-
docker compose -f docker-compose.test.yml up -d --wait --timeout 300
106+
docker compose -f docker-compose.test.yml up -d --build --wait --timeout 300
105107
- name: Select and probe integration endpoint
106108
working-directory: sdk-python
107109
run: python scripts/ci/configure-integration-endpoint.py
@@ -113,7 +115,7 @@ jobs:
113115
- name: Teardown
114116
if: always()
115117
working-directory: sdk-python
116-
run: docker compose -f docker-compose.test.yml down -v
118+
run: docker compose -f docker-compose.test.yml down -v --rmi local
117119

118120
target-branch-qualification:
119121
name: Target branch qualification

‎CHANGELOG.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
66

77
## [Unreleased]
88

9-
The Python SDK advances to the synchronized Durable Workflow `2.0.0-rc.1`
10-
product train. PyPI displays the normalized PEP 440 identity `2.0.0rc1`. Earlier SDK
9+
The Python SDK advances to the synchronized Durable Workflow `2.0.0-rc.3`
10+
product train. PyPI displays the normalized PEP 440 identity `2.0.0rc3`. Earlier SDK
1111
versions remain historical releases and are not alternate supported 2.0
1212
baselines; no prerelease compatibility shim is provided.
1313

1414
### Added
15+
- The optimized `fastavro` payload path now uses the fixed recursive
16+
`durable_workflow.protocol.Value` schema with explicit named branches and
17+
Avro single-object framing. Cross-language fixtures cover every branch,
18+
schema evolution, nested values, and stable policy failures.
19+
- A repeatable benchmark reports JSON, legacy-wrapper, and fixed-schema raw and
20+
HTTP-envelope sizes plus adapter/encode/decode latency over the shared
21+
checked-in corpus, with an enforced production-path regression budget
22+
calibrated with explicit headroom.
1523
- Schedule listing now exposes server-side status, workflow-type, visibility-
1624
query, page-size, and continuation-token filters on async and sync clients.
1725
`ScheduleListError` preserves typed filter and cursor refusal evidence.

‎README.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@
33
A Python SDK for the [Durable Workflow server](https://github.com/durable-workflow/server). Speaks the server's language-neutral HTTP/JSON worker protocol — no PHP runtime required.
44

55
Status: **Beta** — this SDK is part of the synchronized Durable Workflow
6-
`2.0.0-rc.1` product train. Core features include workflows, activities,
6+
`2.0.0-rc.3` product train. Core features include workflows, activities,
77
schedules, signals, timers, child workflows, continue-as-new, side effects,
88
version markers, worker-applied accepted updates, replay verification, the
99
in-process `WorkflowEnvironment` test harness, and invocable activity carriers.
1010

1111
## Install
1212

1313
```bash
14-
pip install durable-workflow==2.0.0-rc.1
14+
pip install durable-workflow==2.0.0-rc.3
1515
```
1616

1717
Or for development:
@@ -429,15 +429,19 @@ quiet_client = Client(
429429

430430
## Avro payload type boundaries
431431

432-
The default Avro codec uses a generic JSON wrapper so PHP, Python, and other
433-
workers can exchange the same wire format. It preserves JSON-native values:
434-
`None`, booleans, numbers, strings, lists, and dictionaries with string keys.
432+
The default Avro codec uses the fixed recursive
433+
`durable_workflow.protocol.Value` schema and standard single-object framing.
434+
It preserves `None`, booleans, signed 64-bit integers, finite doubles, bytes,
435+
UTF-8 strings, lists, and dictionaries with string keys as distinct branches.
436+
Unknown schema fingerprints fail with `unsupported_payload_schema`; the codec
437+
never guesses or silently falls back to JSON.
435438

436439
Class-carrying values are not encoded with type metadata. Convert pydantic
437440
models, attrs classes, dataclasses, pendulum values, `datetime` / `date` /
438441
`time`, `UUID`, `Decimal`, and plain `Enum` values to explicit dictionaries or
439442
scalars before passing them to the SDK. `IntEnum` and `StrEnum` encode because
440-
they are JSON scalar subclasses, but they decode as `int` and `str`.
443+
they are Python scalar subclasses selected as Avro `LongValue` and
444+
`StringValue`, but they decode as `int` and `str`.
441445
`OrderedDict` decodes as a plain `dict`.
442446

443447
Use `to_avro_payload_value(...)` when a rich value should enter durable
@@ -581,7 +585,7 @@ Full documentation is available at
581585

582586
## Compatibility
583587

584-
SDK version `2.0.0-rc.1` is supported with server `2.0.0-rc.1`. The server
588+
SDK version `2.0.0-rc.3` is supported with server `2.0.0-rc.3`. The server
585589
must advertise these protocol manifests from `GET /api/cluster/info`:
586590

587591
- `control_plane.version: "2"`

‎benchmarks/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Avro Value benchmark
2+
3+
`python benchmarks/avro_value.py --enforce` loads the checked-in medium corpus
4+
at `schema/avro-value-benchmark-v1.json` (SHA-256
5+
`588771404977f2a95fe7d8969c24a15e1c7dd78fe498af9aa2406f82be54b666`).
6+
The bytes sentinel is adapted only for the fixed typed path; compact JSON and
7+
the removed wrapper use the corpus's documented JSON representation.
8+
9+
A release qualification run in the Linux Python 3 worker measured about 65 µs
10+
to encode and 43 µs to decode through `fastavro`. The enforced 125/100 µs
11+
defaults allow about 2x shared-runner variance while remaining far below the
12+
slow recursive reference path. Set `AVRO_VALUE_ENCODE_BUDGET_US` and
13+
`AVRO_VALUE_DECODE_BUDGET_US` to calibrate a different qualification runner.

‎benchmarks/avro_value.py‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
#!/usr/bin/env python3
2+
"""Repeatable Avro Value protocol size and latency benchmark.
3+
4+
The benchmark compares compact JSON, the prerelease JSON-in-Avro wrapper, and
5+
the fixed typed Value schema. The production-path regression budget covers the
6+
fixed codec only; the old wrapper remains here solely as migration evidence.
7+
"""
8+
from __future__ import annotations
9+
10+
import argparse
11+
import base64
12+
import hashlib
13+
import io
14+
import json
15+
import os
16+
import statistics
17+
import time
18+
from collections.abc import Callable
19+
from pathlib import Path
20+
from typing import Any
21+
22+
from fastavro import parse_schema, schemaless_reader, schemaless_writer
23+
24+
from durable_workflow._avro import (
25+
decode,
26+
encode,
27+
)
28+
29+
OLD_WRAPPER_SCHEMA = parse_schema(
30+
{
31+
"type": "record",
32+
"name": "Payload",
33+
"namespace": "durable_workflow.protocol.prerelease",
34+
"fields": [{"name": "json", "type": "string"}, {"name": "version", "type": "int"}],
35+
}
36+
)
37+
CORPUS_PATH = Path(__file__).parents[1] / "schema" / "avro-value-benchmark-v1.json"
38+
CORPUS_BYTES = CORPUS_PATH.read_bytes()
39+
CORPUS = json.loads(CORPUS_BYTES)
40+
JSON_SAMPLE: dict[str, Any] = CORPUS["value"]
41+
42+
43+
def _adapt_bytes(value: Any) -> Any:
44+
if isinstance(value, dict) and list(value) == ["$avro_bytes"]:
45+
return base64.b64decode(value["$avro_bytes"], validate=True)
46+
if isinstance(value, dict):
47+
return {key: _adapt_bytes(item) for key, item in value.items()}
48+
if isinstance(value, list):
49+
return [_adapt_bytes(item) for item in value]
50+
return value
51+
52+
53+
SAMPLE: dict[str, Any] = _adapt_bytes(JSON_SAMPLE)
54+
55+
56+
def _json_value(value: Any) -> Any:
57+
if isinstance(value, bytes):
58+
return {"$type": "bytes", "base64": base64.b64encode(value).decode("ascii")}
59+
if isinstance(value, list):
60+
return [_json_value(item) for item in value]
61+
if isinstance(value, dict):
62+
return {key: _json_value(item) for key, item in value.items()}
63+
return value
64+
65+
66+
def _compact_json(value: Any) -> bytes:
67+
return json.dumps(_json_value(value), ensure_ascii=False, separators=(",", ":")).encode()
68+
69+
70+
def _old_encode(value: Any) -> bytes:
71+
datum = {"json": _compact_json(value).decode(), "version": 1}
72+
stream = io.BytesIO()
73+
stream.write(b"\x00")
74+
schemaless_writer(stream, OLD_WRAPPER_SCHEMA, datum)
75+
return stream.getvalue()
76+
77+
78+
def _old_decode(payload: bytes) -> Any:
79+
datum = schemaless_reader(io.BytesIO(payload[1:]), OLD_WRAPPER_SCHEMA)
80+
return json.loads(datum["json"])
81+
82+
83+
def _measure(action: Callable[[], Any], iterations: int) -> float:
84+
samples: list[float] = []
85+
for _ in range(5):
86+
started = time.perf_counter_ns()
87+
for _ in range(iterations):
88+
action()
89+
samples.append((time.perf_counter_ns() - started) / iterations / 1_000)
90+
return statistics.median(samples)
91+
92+
93+
def _http_envelope_size(blob: str) -> int:
94+
return len(
95+
json.dumps({"codec": "avro", "blob": blob}, separators=(",", ":")).encode()
96+
)
97+
98+
99+
def main() -> int:
100+
parser = argparse.ArgumentParser()
101+
parser.add_argument("--iterations", type=int, default=1_000)
102+
parser.add_argument("--enforce", action="store_true")
103+
args = parser.parse_args()
104+
105+
json_bytes = _compact_json(JSON_SAMPLE)
106+
old_payload = _old_encode(JSON_SAMPLE)
107+
typed_blob = encode(SAMPLE)
108+
typed_payload = base64.b64decode(typed_blob)
109+
110+
results = {
111+
"implementation": "fastavro",
112+
"corpus": {
113+
"schema": CORPUS["schema"],
114+
"case": CORPUS["case"],
115+
"sha256": hashlib.sha256(CORPUS_BYTES).hexdigest(),
116+
},
117+
"iterations": args.iterations,
118+
"sizes_bytes": {
119+
"plain_json": {
120+
"raw": len(json_bytes),
121+
"http_envelope": len(
122+
json.dumps(
123+
{"codec": "json", "blob": json_bytes.decode()},
124+
separators=(",", ":"),
125+
).encode()
126+
),
127+
},
128+
"old_json_wrapper": {
129+
"raw_datum": len(old_payload) - 1,
130+
"framed": len(old_payload),
131+
"http_envelope": _http_envelope_size(base64.b64encode(old_payload).decode()),
132+
},
133+
"fixed_typed_value": {
134+
"raw_datum": len(typed_payload) - 10,
135+
"single_object": len(typed_payload),
136+
"http_envelope": _http_envelope_size(typed_blob),
137+
},
138+
},
139+
"latency_us": {
140+
"plain_json_encode": _measure(
141+
lambda: _compact_json(JSON_SAMPLE), args.iterations
142+
),
143+
"plain_json_decode": _measure(lambda: json.loads(json_bytes), args.iterations),
144+
"old_json_wrapper_encode": _measure(
145+
lambda: _old_encode(JSON_SAMPLE), args.iterations
146+
),
147+
"old_json_wrapper_decode": _measure(
148+
lambda: _old_decode(old_payload), args.iterations
149+
),
150+
"fixed_typed_value_encode": _measure(lambda: encode(SAMPLE), args.iterations),
151+
"fixed_typed_value_decode": _measure(lambda: decode(typed_blob), args.iterations),
152+
},
153+
}
154+
print(json.dumps(results, indent=2, sort_keys=True))
155+
156+
if not args.enforce:
157+
return 0
158+
159+
encode_budget = float(os.getenv("AVRO_VALUE_ENCODE_BUDGET_US", "125"))
160+
decode_budget = float(os.getenv("AVRO_VALUE_DECODE_BUDGET_US", "100"))
161+
failed = (
162+
results["latency_us"]["fixed_typed_value_encode"] > encode_budget
163+
or results["latency_us"]["fixed_typed_value_decode"] > decode_budget
164+
)
165+
if failed:
166+
print(
167+
"Avro Value production-path regression budget exceeded: "
168+
f"encode <= {encode_budget:g} us, decode <= {decode_budget:g} us."
169+
)
170+
return 1
171+
return 0
172+
173+
174+
if __name__ == "__main__":
175+
raise SystemExit(main())

‎docker-compose.test.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# Spins up a Durable Workflow server stack for integration tests.
22
#
33
# Usage:
4-
# docker compose -f docker-compose.test.yml up -d --wait
4+
# docker compose -f docker-compose.test.yml up -d --build --wait
55
# DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080 pytest tests/integration/ -v
6-
# docker compose -f docker-compose.test.yml down -v
6+
# docker compose -f docker-compose.test.yml down -v --rmi local
77

88
services:
99
bootstrap:

‎pyproject.toml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "durable-workflow"
7-
version = "2.0.0-rc.1"
7+
version = "2.0.0-rc.3"
88
description = "Python SDK for the Durable Workflow server (language-neutral HTTP protocol)"
99
readme = "README.md"
1010
requires-python = ">=3.10"
@@ -32,7 +32,7 @@ classifiers = [
3232
]
3333
dependencies = [
3434
"httpx>=0.27",
35-
"avro>=1.12,<2",
35+
"fastavro>=1.12.2,<2",
3636
]
3737

3838
[project.optional-dependencies]
@@ -64,9 +64,9 @@ durable-workflow-replay-conformance = "durable_workflow.replay_conformance:main"
6464
durable-workflow-workflow-updates-conformance = "durable_workflow.workflow_updates_conformance:main"
6565

6666
[tool.durable-workflow]
67-
product-train = "2.0.0-rc.1"
68-
registry-version = "2.0.0rc1"
69-
supported-server-versions = "2.0.0-rc.1"
67+
product-train = "2.0.0-rc.3"
68+
registry-version = "2.0.0rc3"
69+
supported-server-versions = "2.0.0-rc.3"
7070
worker-protocol-version = "1.1"
7171
control-plane-version = "2"
7272

@@ -86,9 +86,9 @@ strict = true
8686
packages = ["durable_workflow"]
8787
mypy_path = "src"
8888

89-
# apache/avro does not ship type stubs; treat it as Any.
89+
# fastavro does not ship complete type stubs; treat it as Any.
9090
[[tool.mypy.overrides]]
91-
module = ["avro", "avro.*"]
91+
module = ["fastavro", "fastavro.*"]
9292
ignore_missing_imports = true
9393

9494
[tool.ruff]
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"schema": "durable_workflow.protocol.Value.benchmark/v1",
3+
"case": "medium",
4+
"value": {
5+
"workflow": "benchmark",
6+
"attempt": 7,
7+
"ratio": 7.0,
8+
"enabled": true,
9+
"note": "typed value",
10+
"blob": {
11+
"$avro_bytes": "AP8="
12+
},
13+
"items": [
14+
null,
15+
false,
16+
-9,
17+
3.25,
18+
"nested",
19+
{
20+
"region": "us-east-1",
21+
"retries": [1, 2, 3]
22+
}
23+
]
24+
}
25+
}

0 commit comments

Comments
 (0)