Establish scaffolding for Service Bus

[sigurdp]

Added initial Azure Service Bus infrastructure for Webviz, including a skeleton Python worker, message publishing from the primary backend, and local Service Bus emulator support.

Changes:

• Add Service Bus destination helpers for Radix environments and local development.
• Added helper class for sending messages, MessageBus
• Add the pyworker package that has skeleton implementation for message processing. Currently only a single dev test message is handled.
• Configure the Radix component, identities, secrets, and queue-based scaling.
• Extended and matured TaskMetaTracker
• Update backend dependencies
• Several minor refactors and adaptions

Note:
The worker currently dispatches only the DEV_TEST operation. The tracked-task runner is scaffolding and is not yet connected to message dispatch.

[Copilot]

🟡 Changes recommended

Startup compatibility, atomic metadata updates, settlement handling, and publishing limits need correction.

4 open findings

• Make task state updates atomic to prevent partial Redis records · New
• Recordings workflow lacks required Service Bus and payload configuration · New
• Cap and restrict the production-accessible dev load endpoint · New
• Prevent completion failures from dead-lettering successful work · New

What changed in this PR

Establishes Azure Service Bus scaffolding for backend publishing and Python worker processing, with local emulator support and Radix deployment configuration.

Changes:

• Adds message publishing, schemas, and a worker handling DEV_TEST.
• Introduces task lifecycle tracking and an unconnected tracked-task runner.
• Configures worker deployment, scaling, dependencies, and CI checks.

File	Description
radixconfig.yml	Adds worker deployment, identities, secrets, and scaling.
docker/​sb-emulator/​SbEmulatorConfig.json	Defines the local emulator queue.
docker-compose.yml	Adds worker and messaging environment variables.
docker-compose-service-bus.yml	Adds Service Bus emulator services.
backend_py/​scripts/​pylint-pyworker-all.sh	Adds worker lint checks.
backend_py/​scripts/​poetry-regenerate-lock-files.sh	Extends lockfile regeneration.
backend_py/​scripts/​mypy-pyworker-all.sh	Adds worker type checks.
backend_py/​pyworker/​src/​pyworker/​worker_main.py	Provides the worker entry point.
backend_py/​pyworker/​src/​pyworker/​worker_config.py	Loads worker configuration.
backend_py/​pyworker/​src/​pyworker/​worker_app.py	Manages receiving, concurrency, telemetry, and shutdown.
backend_py/​pyworker/​src/​pyworker/​utils/​worker_logging.py	Adds contextual worker logging.
backend_py/​pyworker/​src/​pyworker/​utils/​message_decryption.py	Adds payload decryption.
backend_py/​pyworker/​src/​pyworker/​utils/​abort_signal.py	Provides cooperative shutdown signaling.
backend_py/​pyworker/​src/​pyworker/​utils/​__init__.py	Initializes the utilities package.
backend_py/​pyworker/​src/​pyworker/​tasks/​dev_test_task.py	Implements the test message handler.
backend_py/​pyworker/​src/​pyworker/​tasks/​__init__.py	Initializes the tasks package.
backend_py/​pyworker/​src/​pyworker/​task_runner.py	Adds tracked-task lifecycle scaffolding.
backend_py/​pyworker/​src/​pyworker/​task_exceptions.py	Defines processing outcome exceptions.
backend_py/​pyworker/​src/​pyworker/​process_message.py	Dispatches messages and handles settlement.
backend_py/​pyworker/​src/​pyworker/​__init__.py	Initializes the worker package.
backend_py/​pyworker/​pyproject.toml	Defines worker dependencies and packaging.
backend_py/​pyworker/​Dockerfile	Builds the worker container.
backend_py/​primary/​pyproject.toml	Declares encryption dependency.
backend_py/​primary/​primary/​routers/​surface/​task_helpers.py	Adapts surface tasks to revised metadata.
backend_py/​primary/​primary/​routers/​dev/​router.py	Adds test message publishing.
backend_py/​primary/​primary/​middleware/​encrypted_redis_session_store.py	Documents Fernet key generation.
backend_py/​primary/​primary/​main.py	Initializes and closes the message bus.
backend_py/​primary/​primary/​config.py	Adds Service Bus configuration.
backend_py/​primary/​poetry.lock	Updates resolved backend dependencies.
backend_py/​libs/​services/​src/​webviz_services/​utils/​task_meta_tracker.py	Expands task states, timestamps, and updates.
backend_py/​libs/​services/​src/​webviz_services/​platform/​message_bus.py	Adds cached, serialized queue publishing.
backend_py/​libs/​services/​src/​webviz_services/​platform/​azure_credentials.py	Shares the Azure credential factory.
backend_py/​libs/​services/​src/​webviz_services/​platform/​__init__.py	Initializes the platform package.
backend_py/​libs/​services/​pyproject.toml	Adds Azure messaging dependencies.
backend_py/​libs/​server_schemas/​src/​webviz_server_schemas/​pyworker/​messages.py	Defines worker message schemas.
backend_py/​libs/​server_schemas/​src/​webviz_server_schemas/​pyworker/​__init__.py	Initializes the worker schemas package.
backend_py/​libs/​core_utils/​src/​webviz_core_utils/​service_bus_destination.py	Resolves environment-specific messaging destinations.
backend_py/​libs/​core_utils/​src/​webviz_core_utils/​radix_utils.py	Documents Radix environment helpers.
.github/​workflows/​webviz.yml	Adds worker checks and API-generation configuration.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🟡 Changes recommended

Exceptional worker cleanup and unchecked task-state writes leave processing and tracking correctness unresolved.

1 open finding

• Drain active message tasks when receiving fails · New

4 resolved since last review

• Cap and restrict the production-accessible dev load endpoint
• Recordings workflow lacks required Service Bus and payload configuration
• Make task state updates atomic to prevent partial Redis records
• Prevent completion failures from dead-lettering successful work

Previously missed (1)

In code that hasn't changed since last review

Handle rejected task status updates

backend_py/​pyworker/​src/​pyworker/​task_runner.py:67

The tracker returns False when the task record has expired, been evicted, or been deleted, but every lifecycle write here ignores that result. When this scaffolding is used, it can execute work without recording RUNNING and return successfully without recording SUCCEEDED, allowing completion without a stored outcome. Check all RUNNING and terminal-state writes and raise TaskTrackingError when an update is rejected, including the failure-state branches.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🟡 Changes recommended

Sumo integration-test collection lacks required configuration, and the tracked-task runner ignores rejected metadata updates.

1 open finding

• Add missing Sumo workflow test configuration · New

1 resolved since last review

• Drain active message tasks when receiving fails

Previously missed (1)

In code that hasn't changed since last review

Handle failed task metadata updates in runner

backend_py/​pyworker/​src/​pyworker/​task_runner.py:63

set_state_async() and fail_task_async() return False when task metadata has expired, been evicted, or been deleted. This runner ignores every result: it can start work_fn without metadata and return successfully without recording the outcome. Although the runner is not connected to dispatch yet, this breaks its stated tracking contract. Check the initial and terminal writes and raise TaskTrackingError when a required update fails, rather than treating the transition as successful.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🔵 Needs a closer look

Unresolved correctness issues remain, and production identity and queue-scaling integration require human validation.

2 open findings

• Skip undecodable binary properties during metadata extraction · New
• Raise tracking error when task-state writes are not persisted · New

1 resolved since last review

• Add missing Sumo workflow test configuration

Previously missed (1)

In code that hasn't changed since last review

Support legacy startTimeUtcS when reading task progress

backend_py/​libs/​services/​src/​webviz_services/​utils/​task_meta_tracker.py:201

Existing task hashes store startTimeUtcS, but this reader now only recognizes startedAtUtcS under the same Redis key prefix. Surface tasks remain discoverable by fingerprint for up to 23 hours, yet their progress responses now report 0.0 seconds elapsed. Fall back to the previous timestamp field while those records expire.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.