Idea
NICOS wants the workflow parameters a scan counts against to be defined in the scan script, as the single point of definition. The script would configure and start a workflow, count against its output, and stop it at run end. Discussed with the NICOS team in the 2026-09 interface meeting; this issue records the resulting plan so it can be refined with them. Nothing here is decided.
This changes the premise of ADR 0006. There, NICOS was a passive consumer of jobs the dashboard started, which is why identity had to be made static and why provenance could not be expressed. If NICOS starts the job, it chooses the job number, so it can filter its own results, and provenance lives in the script.
Proposed shape
A dedicated backend instance for NICOS, on its own host, with its own topic namespace for outputs, commands, and heartbeats (raw input topics unchanged). Each backend then has exactly one controlling client, which is the invariant the code already assumes (ADR 0008, WorkflowConfig docstring). Sharing one backend between two clients is what produced every awkward piece so far: the extractor publishing two jobs under one device name, foreign-job adoption, a confirmation gate that warns but cannot know, and the ownership concept that would be needed on configs and heartbeats.
Topic separation alone is not isolation; only a separate process gives NICOS its own failure domain, restart, version pin, and on-call scope.
Scripts pass parameters. WorkflowConfig already needs only a workflow id, a job id (source name plus a job number NICOS mints), and a params dict. Results are keyed by that job id. Acknowledgements exist, so a script can abort before motors move if validation fails. Requirements: parameter models must reject unknown fields for external callers (to be verified), and scripts pin instrument/name/version, so we own a compatibility promise toward scripts.
A lightweight workflow profile. The instance exposes a small, mostly static set of scan-feedback workflows per instrument: counts and simple derived scalars, no images, no reduction. Keeps the duplicated raw consumption and compute small. Discovery is a generated, committed artifact per release (workflow ids, parameter JSON schema, aux source roles, defaults), in the same generated-and-drift-tested shape as device_contract.yaml. Not needed for a first version with a static set.
NICOS does its own plotting of scan feedback. The dashboard does not show what a scan counts against; that would be the side-by-side duplication we want to avoid. A user asking why NICOS's total differs from the dashboard's gets "NICOS counts what the script says", which only holds if parameters are visible in the script rather than hidden in a default.
Second step, after the replica works
Dynamic raw-topic subscription. NICOS runs one or a few jobs at a time, unlike the dashboard. An instance that assigns raw topics only while a job needs them pays nothing for idle sources. That is what makes it safe to drop sharding and run a single process on a high-rate instrument (without it, one process consuming everything is the shape behind #378 / #1264). Order: replica with production sharding first (reuses the deployment), then dynamic subscription, then collapse to one process. Watch the batcher with sources appearing and disappearing mid-run.
Not doing: Kafka consumer-group coordination of job assignment across workers. Consumer groups balance partitions, not jobs; a rebalance mid-scan would wipe or require handoff of in-process accumulator state; acks and heartbeats would have to survive ownership changes. The workload does not need it.
What this retires
StreamKind.LIVEDATA_NICOS_DATA, DeviceExtractor, and the device contract as a wire mechanism (device names may live on as NICOS-side script defaults).- Dashboard derived-device gate and overview.
- The single-client assumptions stay true per deployment, so ADR 0008 needs no amendment; ADR 0006 would be superseded.
Open questions for the NICOS side
- Which workflows and parameters do the first scan scripts need, per instrument?
- Job lifetime backstop: stop by script, or tie NICOS-started jobs to run stop from the file writer (the backend already listens to it)?
- Who operates the NICOS instance, and does it run a pinned esslivedata version?
- Restart behaviour: NICOS holds the config, so it can re-issue the start after a backend restart and repeat the scan point. Is that the intended recovery?
On our side before a first test
- Namespace flag for
stream_kind_to_topic and the CLI.
- Workflow registry profile restricted to scan-feedback workflows.
- Verify strict parameter validation for external callers.
- Topic creation with ECDC.
Related: #445, #737, #856, #1051, #1242, ADR 0006, ADR 0008.
Idea
NICOS wants the workflow parameters a scan counts against to be defined in the scan script, as the single point of definition. The script would configure and start a workflow, count against its output, and stop it at run end. Discussed with the NICOS team in the 2026-09 interface meeting; this issue records the resulting plan so it can be refined with them. Nothing here is decided.
This changes the premise of ADR 0006. There, NICOS was a passive consumer of jobs the dashboard started, which is why identity had to be made static and why provenance could not be expressed. If NICOS starts the job, it chooses the job number, so it can filter its own results, and provenance lives in the script.
Proposed shape
A dedicated backend instance for NICOS, on its own host, with its own topic namespace for outputs, commands, and heartbeats (raw input topics unchanged). Each backend then has exactly one controlling client, which is the invariant the code already assumes (ADR 0008,
WorkflowConfigdocstring). Sharing one backend between two clients is what produced every awkward piece so far: the extractor publishing two jobs under one device name, foreign-job adoption, a confirmation gate that warns but cannot know, and the ownership concept that would be needed on configs and heartbeats.Topic separation alone is not isolation; only a separate process gives NICOS its own failure domain, restart, version pin, and on-call scope.
Scripts pass parameters.
WorkflowConfigalready needs only a workflow id, a job id (source name plus a job number NICOS mints), and a params dict. Results are keyed by that job id. Acknowledgements exist, so a script can abort before motors move if validation fails. Requirements: parameter models must reject unknown fields for external callers (to be verified), and scripts pininstrument/name/version, so we own a compatibility promise toward scripts.A lightweight workflow profile. The instance exposes a small, mostly static set of scan-feedback workflows per instrument: counts and simple derived scalars, no images, no reduction. Keeps the duplicated raw consumption and compute small. Discovery is a generated, committed artifact per release (workflow ids, parameter JSON schema, aux source roles, defaults), in the same generated-and-drift-tested shape as
device_contract.yaml. Not needed for a first version with a static set.NICOS does its own plotting of scan feedback. The dashboard does not show what a scan counts against; that would be the side-by-side duplication we want to avoid. A user asking why NICOS's total differs from the dashboard's gets "NICOS counts what the script says", which only holds if parameters are visible in the script rather than hidden in a default.
Second step, after the replica works
Dynamic raw-topic subscription. NICOS runs one or a few jobs at a time, unlike the dashboard. An instance that assigns raw topics only while a job needs them pays nothing for idle sources. That is what makes it safe to drop sharding and run a single process on a high-rate instrument (without it, one process consuming everything is the shape behind #378 / #1264). Order: replica with production sharding first (reuses the deployment), then dynamic subscription, then collapse to one process. Watch the batcher with sources appearing and disappearing mid-run.
Not doing: Kafka consumer-group coordination of job assignment across workers. Consumer groups balance partitions, not jobs; a rebalance mid-scan would wipe or require handoff of in-process accumulator state; acks and heartbeats would have to survive ownership changes. The workload does not need it.
What this retires
StreamKind.LIVEDATA_NICOS_DATA,DeviceExtractor, and the device contract as a wire mechanism (device names may live on as NICOS-side script defaults).Open questions for the NICOS side
On our side before a first test
stream_kind_to_topicand the CLI.Related: #445, #737, #856, #1051, #1242, ADR 0006, ADR 0008.