Skip to content

Commit 0ed44b8

Browse files
authored
Merge pull request #162: Document separate worker processes
Document separate workflow and activity worker processes
2 parents 101c591 + 2310f1d commit 0ed44b8

6 files changed

Lines changed: 395 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ for the complete cross-SDK contract.
9999
- [Python SDK guide](https://durable-workflow.com/docs/2.0/polyglot/python/)
100100
- [Complete SDK reference](docs/sdk-reference.md)
101101
- [Runnable examples](examples/)
102+
- [Separate workflow and activity worker processes](docs/guides/separate-workers.md)
102103
- [Symmetric SDK playground](https://github.com/durable-workflow/sample-app#symmetric-sdk-playground)
103104

104105
## Runtime choices

‎docs/guides/separate-workers.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Separate workflow and activity processes
2+
3+
Run workflow coordination separately from remote activity callbacks when they
4+
need different CPU, memory or concurrency limits. A `Worker` polls workflow
5+
tasks only when it has workflow handlers, and activity tasks only when it has
6+
activity handlers. Select the handler lists to separate roles. Both concurrency
7+
settings must remain positive; setting one to zero is not a way to disable it.
8+
9+
The [complete example](https://github.com/durable-workflow/sdk-python/blob/main/examples/separate_workers.py)
10+
has workflow-only, activity-only, start and inspection commands. It uses the
11+
normal installed SDK, without importing the repository's `src` directory.
12+
13+
## Install and connect
14+
15+
Use a POSIX shell on Linux, macOS or WSL with Python 3.10 or newer. From a checkout
16+
of the SDK repository:
17+
18+
```bash
19+
python -m venv .venv
20+
. .venv/bin/activate
21+
python -m pip install 'durable-workflow==2.6.0'
22+
export SERVER_URL=http://localhost:8080
23+
export WORKFLOW_NAMESPACE=default
24+
export WORKFLOW_TOKEN=your-local-development-token
25+
```
26+
27+
The qualification baseline is published Server **2.9.5** and Python SDK
28+
**2.6.0**. Start Server using its [self-hosted installation guide](https://github.com/durable-workflow/server#run-server).
29+
Its API, queue worker and scheduler must be running and `/api/ready` must be
30+
ready. The Python processes talk to Server over HTTP and do not connect to its
31+
database. `SERVER_URL` is the origin without `/api`. For Cloud, use the complete
32+
provisioned namespace runtime URL and credentials instead.
33+
34+
The local example uses one development token. In a deployed application, set
35+
`WORKFLOW_TOKEN` separately for each process: a worker credential for worker
36+
commands, a client credential for `start`, and an operator credential for
37+
`inspect`. Do not copy an operator token into workers to make inspection work.
38+
39+
## Start two workers
40+
41+
Activate the same environment and set the connection variables in each terminal.
42+
Run the workflow role in the first terminal:
43+
44+
```bash
45+
python examples/separate_workers.py workflow --worker-id python-workflows-1 --slots 4
46+
```
47+
48+
Run the activity role in the second:
49+
50+
```bash
51+
python examples/separate_workers.py activity --worker-id python-activities-1 --slots 2
52+
```
53+
54+
Every live process needs its own worker ID. The two registrations use the same
55+
`python-split` task queue and namespace, with different supported types:
56+
57+
| Worker | Workflow types | Activity types |
58+
| --- | --- | --- |
59+
| `python-workflows-1` | `examples.split-workers.greeter` | empty |
60+
| `python-activities-1` | empty | `examples.split-workers.greet` |
61+
62+
In a third terminal, inspect the actual roster, then start a workflow:
63+
64+
```bash
65+
python examples/separate_workers.py inspect
66+
python examples/separate_workers.py start --name world
67+
python examples/separate_workers.py start --name retry --retry-once
68+
```
69+
70+
The final results are `hello, world` and `hello, retry`. The activity process
71+
prints its worker ID, process ID and attempt number. The retry example fails
72+
attempt 1 deliberately; Server schedules attempt 2 under the activity's recorded
73+
two-attempt policy. The start command only submits and observes. It does not run
74+
either handler itself, and its 30-second wait is not a cancellation request.
75+
76+
The roster reports both numeric slot settings even though each process has only
77+
one handler family. Supported type lists determine which family it serves.
78+
Do not count the inactive family's positive setting as usable handler capacity.
79+
80+
## Route activities to a distinct queue
81+
82+
The shared-queue example separates processes by supported types. It does not
83+
create a separate activity queue. For an explicit queue split, start workers
84+
with matching queue names:
85+
86+
```bash
87+
# Workflow terminal
88+
python examples/separate_workers.py workflow --worker-id python-workflows-orders --queue orders
89+
# Activity terminal
90+
python examples/separate_workers.py activity --worker-id python-activities-heavy --queue heavy
91+
# Client terminal
92+
python examples/separate_workers.py start --queue orders --activity-queue heavy --retry-once
93+
```
94+
95+
The workflow passes `queue="heavy"` to `ctx.schedule_activity(...)`. With no
96+
override, the activity uses the workflow's queue. An activity worker on `heavy`
97+
will not serve work scheduled to `orders`. Namespace, queue and supported type
98+
must all match. Build compatibility adds another routing constraint during
99+
[worker rollouts](https://durable-workflow.com/docs/2.0/polyglot/worker-build-id-rollout/).
100+
101+
## Size and stop each process
102+
103+
`--slots` changes only that role's `max_concurrent_*_tasks`. Set container or
104+
process CPU and memory limits independently. Activity slots must also fit the
105+
database, HTTP connection pool, downstream rate limits and callback memory
106+
budget. More processes multiply local capacity; use
107+
[server-side admission budgets](https://durable-workflow.com/docs/2.0/polyglot/task-queue-admission/)
108+
when the whole namespace or queue needs a ceiling.
109+
110+
An `async def` activity must yield while waiting. Blocking CPU or synchronous
111+
I/O inside it blocks that process's event loop, including polling and heartbeat
112+
work. Use suitable nonblocking libraries or bounded executors/processes for the
113+
workload; a separate activity process does not remove downstream contention.
114+
115+
Press Ctrl-C in each worker terminal, or send SIGTERM to each worker process.
116+
The example awaits `Worker.stop()` and its run task. It stops polling, drains
117+
accepted work within its 10-second shutdown budget, and deregisters. Give the
118+
process supervisor a termination grace period longer than this budget plus
119+
HTTP cleanup time. Errors during deregistration or drain cause a nonzero exit
120+
and must be investigated. Stopping the client does not stop workers or cancel
121+
the workflow. A later `inspect` should no longer list successfully stopped
122+
worker IDs; a missing process alone is not proof of successful deregistration.
123+
124+
## Local resources and recovery
125+
126+
- Ordinary `ctx.schedule_activity(...)` calls are remote activity tasks. They
127+
can be picked up by a compatible replacement activity worker after normal
128+
lease/retry handling. Application side effects still need idempotency.
129+
- Opted-in local activities execute in the **workflow worker**. Register their
130+
callbacks on that worker; an activity-only process cannot supply them. They
131+
consume workflow-process resources and need their documented durable local
132+
activity and cancellation capabilities.
133+
- Worker sessions intentionally bind work to the session owner while its lease
134+
is active. Process separation does not make an in-memory client, file handle
135+
or session durable or migrate it to any available process.
136+
- Sticky history caches belong to workflow workers and are optional. Eviction
137+
or process replacement uses cold replay from Server history. Keep correctness
138+
independent of cache contents.
139+
140+
See [Worker lifecycle and shutdown](../reference/worker.md),
141+
[local activities and sessions](../sdk-reference.md#local-activities), and
142+
[sticky execution](sticky-execution.md) for the respective contracts.

‎docs/reference/worker.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Worker
22

3+
See [Separate workflow and activity processes](../guides/separate-workers.md)
4+
for a runnable deployment example with distinct registrations, independent
5+
slots, remote activity retries and signal-driven shutdown.
6+
37
## Lifecycle and shutdown
48

59
`Worker.run()` and `Worker.run_until()` register through the worker protocol

‎examples/separate_workers.py‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
"""Run workflow and remote activity handlers in separate processes.
2+
3+
See docs/guides/separate-workers.md for installation and three-terminal usage.
4+
"""
5+
from __future__ import annotations
6+
7+
import argparse
8+
import asyncio
9+
import json
10+
import logging
11+
import os
12+
import signal
13+
from uuid import uuid4
14+
15+
from durable_workflow import ActivityRetryPolicy, Client, Worker, activity, workflow
16+
17+
WORKFLOW_TYPE = "examples.split-workers.greeter"
18+
ACTIVITY_TYPE = "examples.split-workers.greet"
19+
20+
21+
@activity.defn(name=ACTIVITY_TYPE)
22+
async def greet(name: str, retry_once: bool) -> str:
23+
info = activity.context().info
24+
print(json.dumps({
25+
"event": "activity_attempt", "worker_id": info.worker_id,
26+
"attempt": info.attempt_number, "process_id": os.getpid(),
27+
}), flush=True)
28+
if retry_once and info.attempt_number == 1:
29+
raise RuntimeError("intentional first-attempt failure for the example")
30+
return f"hello, {name}"
31+
32+
33+
@workflow.defn(name=WORKFLOW_TYPE)
34+
class Greeter:
35+
def run(self, ctx, name: str, retry_once: bool, activity_queue: str | None):
36+
return (yield ctx.schedule_activity(
37+
ACTIVITY_TYPE,
38+
[name, retry_once],
39+
queue=activity_queue,
40+
retry_policy=ActivityRetryPolicy(max_attempts=2, initial_interval_seconds=1),
41+
start_to_close_timeout=5,
42+
schedule_to_close_timeout=20,
43+
))
44+
45+
46+
def positive_slots(value: str) -> int:
47+
slots = int(value)
48+
if slots < 1:
49+
raise argparse.ArgumentTypeError("slots must be at least 1")
50+
return slots
51+
52+
53+
def parse_args() -> argparse.Namespace:
54+
parser = argparse.ArgumentParser(description=__doc__)
55+
roles = parser.add_subparsers(dest="role", required=True)
56+
for role in ("workflow", "activity"):
57+
worker = roles.add_parser(role, help=f"run only {role} handlers")
58+
worker.add_argument("--worker-id", required=True, help="unique for each live worker process")
59+
worker.add_argument("--queue", default="python-split")
60+
worker.add_argument("--slots", type=positive_slots, default=4)
61+
start = roles.add_parser("start", help="start a workflow and wait up to 30 seconds")
62+
start.add_argument("--queue", default="python-split")
63+
start.add_argument("--name", default="world")
64+
start.add_argument("--retry-once", action="store_true")
65+
start.add_argument("--activity-queue", help="optional explicit queue for the remote activity")
66+
inspect = roles.add_parser("inspect", help="read the registered worker roster (operator credential)")
67+
inspect.add_argument("--queue", default="python-split")
68+
return parser.parse_args()
69+
70+
71+
async def run_worker(client: Client, args: argparse.Namespace) -> None:
72+
workflow_role = args.role == "workflow"
73+
worker = Worker(
74+
client,
75+
task_queue=args.queue,
76+
worker_id=args.worker_id,
77+
workflows=[Greeter] if workflow_role else [],
78+
activities=[] if workflow_role else [greet],
79+
max_concurrent_workflow_tasks=args.slots if workflow_role else 1,
80+
max_concurrent_activity_tasks=1 if workflow_role else args.slots,
81+
poll_timeout=5,
82+
shutdown_timeout=10,
83+
)
84+
shutdown = asyncio.Event()
85+
loop = asyncio.get_running_loop()
86+
for requested_signal in (signal.SIGINT, signal.SIGTERM):
87+
loop.add_signal_handler(requested_signal, shutdown.set)
88+
run_task = asyncio.create_task(worker.run())
89+
shutdown_task = asyncio.create_task(shutdown.wait())
90+
try:
91+
done, _ = await asyncio.wait({run_task, shutdown_task}, return_when=asyncio.FIRST_COMPLETED)
92+
if run_task in done:
93+
await run_task
94+
finally:
95+
shutdown_task.cancel()
96+
await asyncio.gather(shutdown_task, return_exceptions=True)
97+
try:
98+
await worker.stop()
99+
finally:
100+
await run_task
101+
for requested_signal in (signal.SIGINT, signal.SIGTERM):
102+
loop.remove_signal_handler(requested_signal)
103+
print(json.dumps({"event": "worker_stopped", "worker_id": worker.worker_id}), flush=True)
104+
105+
106+
async def main(args: argparse.Namespace) -> None:
107+
server_url = os.environ.get("SERVER_URL")
108+
token = os.environ.get("WORKFLOW_TOKEN")
109+
if not server_url or not token:
110+
raise SystemExit("set SERVER_URL and WORKFLOW_TOKEN; see the separate-workers guide")
111+
async with Client(
112+
server_url, token=token, namespace=os.environ.get("WORKFLOW_NAMESPACE", "default"),
113+
) as client:
114+
if args.role in ("workflow", "activity"):
115+
await run_worker(client, args)
116+
elif args.role == "inspect":
117+
roster = await client.list_workers(task_queue=args.queue)
118+
print(json.dumps([{
119+
"worker_id": worker.worker_id,
120+
"task_queue": worker.task_queue,
121+
"status": worker.status,
122+
"workflow_types": worker.supported_workflow_types,
123+
"activity_types": worker.supported_activity_types,
124+
"workflow_slots": worker.max_concurrent_workflow_tasks,
125+
"activity_slots": worker.max_concurrent_activity_tasks,
126+
} for worker in roster.workers], indent=2))
127+
else:
128+
handle = await client.start_workflow(
129+
workflow_type=WORKFLOW_TYPE,
130+
workflow_id=f"python-split-{uuid4().hex}",
131+
task_queue=args.queue,
132+
input=[args.name, args.retry_once, args.activity_queue],
133+
execution_timeout_seconds=60,
134+
run_timeout_seconds=60,
135+
)
136+
print(json.dumps({"event": "started", "workflow_id": handle.workflow_id, "run_id": handle.run_id}))
137+
result = await handle.result(timeout=30, poll_interval=0.2)
138+
print(json.dumps({"event": "completed", "result": result}), flush=True)
139+
140+
141+
if __name__ == "__main__":
142+
logging.basicConfig(level=logging.INFO)
143+
asyncio.run(main(parse_args()))

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ nav:
7777
- Start here: index.md
7878
- SDK reference: sdk-reference.md
7979
- Sticky execution: guides/sticky-execution.md
80+
- Separate worker processes: guides/separate-workers.md
8081
- Query deadlines: query-task-authority.md
8182
- SDK guide: https://durable-workflow.com/docs/2.0/polyglot/python/
8283
- PyPI: https://pypi.org/project/durable-workflow/

0 commit comments

Comments
 (0)