Wavelength lookup tables as streamed context inputs (ADR 0010/0011)

[SimonHeybrock]

Feeds the wavelength lookup table computed by wavelength_lut_workflow back into the backend workflows that need it, replacing the tables loaded from files at import time. Those files describe a nominal chopper configuration, so running in any other configuration is silently wrong; the LUT workflow already computes the right table from the live chopper cascade, but publishes it as a result nothing consumes. This PR closes that loop for DREAM and LOKI.

The new concept is a workflow output republished as an input stream for other workflows. The NICOS derived-device mirror (ADR 0006) is publish-only and nothing consumes a mirrored topic, so this is the first cross-service feedback edge. ADR 0010 records the design; the commits implement it end to end.

What it does

The LUT workflow keeps its single chopper_cascade source and publishes two tables — one for the detectors, one for the monitors — declared as context outputs on a dedicated topic. Consumers ask for their group's table by inserting one provider, and select their own rows from it.

Two rather than one per component, because a table is a function of distance and event_time_offset alone and carries no component identity: a per-component table is merely a restriction of the same function. What varies across components is only which stretch of beamline must be covered, so components that share a stretch share a table with nothing lost. Two rather than one overall, because a monitor job has no use for the detectors' dense rows, which are the large payload.

The user-facing distance-range parameter is gone. It defaulted to a span covering no instrument correctly, so an operator starting the workflow with defaults could silently blank every detector: a lookup outside the table's range yields NaN, those events fall outside every histogram bin, and the component renders empty with no error anywhere.

Decisions worth pulling out

A table carries rows only where components are. Neither table is a single uniform grid — monitors sit tens to hundreds of metres upstream of the detectors — so a table is a concatenation of uniform blocks: one dense block across the detectors, one per monitor. LOKI's four monitors cost twenty-six rows in total. Empty rows are not free: BIFROST's 155 m span at a 0.1 m resolution is 1550 rows of 286 event-time-offset bins, a 3.5 MB message against a broker's 1 MB default, and 35 MB at 0.01 m, to carry what fits in tens of kilobytes.

The concatenation is deliberately not uniform, and essreduce's interpolator_numba assumes uniformity — it locates a row by dividing, reading the wrong row silently, and only under numba, since the scipy fallback handles an uneven axis correctly. A consumer therefore selects its own block before the table reaches essreduce, matching the Ltotal its graph already computes against the distances the table already carries. Component identity stays off the wire entirely. The boundary between blocks is stated, in a block coord, rather than inferred from where the row spacing jumps: inferring it is a threshold on a float difference, tuned against the padding the table builder happens to add, and it would forbid the producer from ever emitting two blocks that overlap or abut, which two nearby monitors otherwise would.

Ranges are derived, not declared. Each component's range runs essreduce's own DetectorLtotal and MonitorLtotal providers — the very ones the consumer runs at lookup time — so the range and the lookup agree by construction rather than by review. An earlier draft of the ADR had the Ltotal rule declared per component, justified by indirect geometry putting a table "tens of metres" from where it is queried. That magnitude was wrong: the secondary flight path is metres against primary paths of tens to hundreds of metres, so padding absorbs it and the declaration buys exactness the table's distance resolution does not reward. Over-padding costs recompute in the LUT job and nothing else; under-padding is silent.

Motion is declared as an axis range, not a displacement. The geometry artifact stores a live f144-driven transform as an empty NXlog, so the component riding it has no position at all until something supplies an axis value. Instruments declare one AxisRange per moving axis, keyed by NeXus transform path. Both bounds are axis values in the axis's own units, so the transform keeps supplying the direction and sense of the motion: the range comes from evaluating the geometry at the bounds, not from assuming which way along the beam the axis travels. Which components ride an axis stays derived, since a component is affected precisely when the axis appears in its depends_on chain. LOKI's carriage runs from 0 to 15 m, placing the rear bank at 28.5 .. 43.9 m. Translations are bracketed exactly where it matters — pixel positions are affine in the value, so Ltotal is convex and its maximum sits at a corner of the box the bounds span, and padding absorbs the interior minimum a component crossing the sample plane could have. An angle has no such property, so a live rotation axis is refused rather than approximated and its component falls into the no-table case below.

A component that cannot be placed gets no block. beam_monitor_m4 rides its own undeclared axis, so the LUT workflow cannot place it and lays out no rows for it. A group with no placeable component at all publishes no table, so its stream is never declared and a job asking for it fails at creation rather than waiting forever. Where the group is fine but one component is not, a job on that source gates like any other and then fails at its first recompute, reporting its flight path against the table's coverage. An I(Q) aux selection that picks such a monitor is rejected at job creation instead, with an error naming the monitor: there the selector is the thing at fault, and the job's own source may well be placeable.

Coordinate mode stays a parameter on one workflow. Gating every job of such a spec on the table would make time-of-arrival — the mode you fall back to when everything else is broken — depend on an operator-started job that re-emits only on chopper change. ADR 0003 prescribed splitting the spec for exactly this case. That was implemented here and then reverted: the dashboard data plane is keyed by (workflow_id, source_name, output_name), so two specs give the two modes different output identities, a plot cannot follow a mode switch, the operator runs two jobs, and every per-instrument params override is duplicated. Instead the gate is derived from the graph the params build, so nothing states the condition at all: the factory inserts the reassembly provider unconditionally, a TOA job's graph reduces straight from event_time_offset and leaves that provider on a branch no target reaches, and the job therefore requests nothing and gates on nothing. This supersedes ADR 0003's param-dependent-context non-goal, which was a YAGNI call made when the only over-gated stream was an always-on control-system PV.

Both of a reduction's monitor roles read the one monitor table. I(Q) needs a table per sciline Component — the detector plus the incident and transmission monitor roles — and which physical monitor fills a role is a per-job aux selection. Sharing the monitor table dissolves the problem rather than solving it: one stream, and one provider generic in MonitorType that sciline instantiates per role, each instance selecting its rows by that role's own MonitorLtotal. Which monitor fills a role is settled by geometry the job already holds, not by the stream it reads.

This supersedes three earlier shapes: binding the default monitors and raising when the selection differed, which made the aux selector a lie in wavelength mode; binding every candidate monitor to its own synthesized key, leaving the unselected ones dead parameters; and aux-templated stream names (wavelength_lut/{incident_monitor}), where the binding named the aux field and gate resolution rendered it against the job's selection. That last mechanism worked and was the most intricate thing in the PR — declared-versus-resolved names, route derivation expanding a template over an aux field's declared choices to keep subscriptions a superset. It has no user left and is removed, on both sides of the mirror: stream names are plain and fixed at declaration time again, which is what the statically derived Kafka subscriptions and ADR 0006's no-job-identity rule both want.

The gate is derived, not declared

Whether a job reads the table used to be stated twice: the factory inserted the reassembly provider, and the instrument declared a ContextBinding naming the specs, sources and params that gate on it. Two statements of one fact, which can disagree — and the silent direction of the disagreement leaves a job in pending_context forever, waiting for a stream it would never read. The declaration form also could not express a spec reading another group's table without contortion: LOKI's I(Q) had to bind the monitor table against its detector sources, because the monitors arrive as an aux selection and are not the jobs' source names.

The graph already knows. An instrument now only offers a stream under the workflow key it fills (Instrument.offer_context_stream), naming no spec, source or params, and StreamProcessorWorkflow.build keeps a declaration when its key is an ancestor of a target key — the exact condition under which finalize would fail without it. Reachability rather than insertion is what makes this work for a params-dependent mode: a provider the targets cannot reach is pruned, so the factory inserts unconditionally and the params decide. Job creation then reads the result back off the workflow, so the order becomes validate → build → gate; nothing consumed the gating set before the build, so the reordering costs nothing.

The two mechanisms divide cleanly, but not along push-versus-pull, which is where an earlier draft of this PR drew the line. An offer is derivable whenever the graph names the key, so a binding is required in three cases only: the key reaches the graph solely because the binding injects it (a chain patch is never mentioned by the graph, so there is no request to read back), the stream filling a key varies per source (offers are bijective instrument-wide), or the stream is private to one spec. That last one is a routing constraint rather than a semantic one: route derivation runs at startup with no params while an offer is taken up per job from the built graph, so every service hosting any spec must subscribe to every offered stream. The wavelength LUT's own per-chopper setpoints stay bindings for that reason — they are consumed by one workflow on the timeseries service, and offering them would add twelve dead subscriptions each to data_reduction, detector_data and monitor_data on BIFROST.

Motion is therefore not automatically on the binding side. BIFROST's tank and sample rotations are ordinary parameters of the cut workflow, so they are offered too, and the two skip_instrument_contexts() calls that used to restate graph reachability by hand — the detector view sums over banks, the ratemeter is counts-only — are gone with them. Route derivation gathers offered stream names alongside binding names, which it did not need to do while every offer was published on the context topic and dropped again; without it, moving BIFROST silently unsubscribed data_reduction from the motion topic. An offered name is also checked at registration against what anything actually publishes, the check bindings already had: a gate on a name nobody publishes never opens.

ContextBinding.predicate loses its only user and is removed, along with the two-method split that existed only to filter by it. Params validation still moves up into job creation and WorkflowFactory.create still takes the validated model, so there is one validation site rather than two that can drift.

The stream-name-to-key mapping is declared exactly once either way: consuming factories contribute only the reassembly provider. The registration-time conflict check surfaced that Instrument.load_factories was not idempotent — repeated loading appended duplicate bindings whose synthesized chopper-setpoint keys are fresh objects per run — so it now no-ops on a loaded instrument.

Unrelated but adjacent: scripts/visualize_workflows.py called WorkflowFactory.create without params and so failed to create every workflow it tried to render, swallowing the TypeError per workflow. Fixed here since the same call site needed the new argument.

Scope

DREAM and LOKI migrate; they are the only instruments offering wavelength mode today. The detector- and monitor-view factories lose their lookup-table filename argument outright rather than defaulting it, so a view has no file path left to fall back to, and LOKI's I(Q) reduction migrates with them. This also removes a pooch download from both instruments' startup path. What remains of LookupTableFilename is set explicitly by the unmigrated BIFROST and ESTIA reduction pipelines at their own call sites — an input, not a fallback.

Wavelength mode is also no longer offered where it cannot work. Every logical view on every instrument previously offered it in the UI and raised at job start, since they run on InstrumentDetectorSource, which carries no geometry. Those specs now use a time-of-arrival-only params model, and a test pins that any spec accepting wavelength can actually build a workflow in that mode.

Known gaps

beam_monitor_m4 has no wavelength mode until its axis range is declared. The identity coord and consumer clearing described in ADR 0010 are not implemented yet (#1248), so a new table does not currently reset accumulated statistics. The standing limitations the ADR records — nothing guarantees the LUT job is running, a backend restart loses the table, and the gate protects startup rather than steady state — all still apply.

Test plan

• Full fast suite
• Full suite including slow tests
• End-to-end test running the real chain on DREAM: the LUT job computes both tables, they are extracted, serialized to da00, ingested back through the Kafka route, and reduced with by a wavelength-mode monitor job that selects its own block, with no file anywhere
• Service-level test covering LOKI I(Q), created through JobFactory, gating on both tables whichever monitors it selects, its monitor roles selecting their blocks by flight path, and a selection whose monitor has no block being rejected
• Unit tests for the derivation itself: a declared stream is taken up and wired when the targets reach its key, ignored when the provider is unreachable, and a key the graph needs that nobody declared fails the build
• BIFROST's rotation gate, now derived rather than declared: the three Q-maps gate on both rotation streams, the detector view and the ratemeter on neither
• DREAM detector_projection jobs created in both modes: wavelength gates on wavelength_lut/detectors, TOA gates on nothing, ROI aux routed to the job in both and gating in neither
• Dashboard launches against DREAM
• Start the lookup-table workflow and confirm the detector and monitor tables appear, the monitor one plotted as "Overlay 1D" (its distance axis is not uniform)
• Confirm a wavelength-mode view sits in WAITING FOR CONTEXT until the LUT job emits, then reduces
• Confirm a TOA-mode view starts immediately with no LUT job running

[nvaytet]

I didn't look at everything yet, but I have a preliminary question.

[nvaytet]

I find this ADR much too long. When reading it, it's difficult to make sure we have not missed important bits, as one's focus drifts away half way through.
My guess is that it comes from Claude's OCD to leave no stone un-turned, but there is no way I can fit all this into my head in one go.
ADRs used to be relatively brief and to the point. I don't find keeping in the docs such ADRs useful because no one will read them (apart from AI agents maybe? or maybe that was the point, keep them for a future agent?)

[SimonHeybrock]

I agree. I had reviewed the full initial version, but then went through several update rounds where I just looked at amendments, each of which grew it. I'll see how it can be trimmed down (I suppose a lot of the details are just documented in the code itself).

[nvaytet]

In scipp/ess#603 we make one table per component. I don't see an issue in making one for each monitor, apart from maybe displaying the results?
But it would be nice if the workflow here was functioning in the same way as the GenericUnwrapWorkflow?

[SimonHeybrock]

Did that in an earlier version of the branch, but it came with several problems.

[nvaytet]

Why are we concatenating if this causes special handling later? Can't we make one uniform table per component and then not have to worry beyond that? Or is there an issue with publishing many tables?

If we want to publish just a single thing, can we publish a DataGroup? or maybe we have no schema for a DataGroup on kafka?

[SimonHeybrock]

Per-component tables caused complications when wiring back into the workflows. I did this on an earlier version of the branch. It also complicated plotting of the tables.

Indeed, we do not have a schema fro DataGroup.

[nvaytet]

Per-component tables caused complications when wiring back into the workflows.

Can you elaborate a little on what were the problems?

[SimonHeybrock]

Three things, all of which the branch went through before the block layout.

Wiring it into LOKI's I(Q). Two mechanisms meet badly there, so a bit of setup:

• When you start an I(Q) job you pick, from dropdowns, which physical monitor acts as the incident monitor and which as the transmission monitor -- LOKI has four candidates. These are "aux selections": the job's own source is a detector bank, and the monitors are auxiliary inputs chosen per job.
• A streamed context input is declared the other way round: a ContextBinding says, at import time, "this workflow's key X is filled by the stream named Y", and a job with such a binding waits (pending_context) until that stream arrives. It is a property of the workflow spec, fixed before any job exists.

With one table per component, the stream a job needs -- wavelength_lut/loki_monitor_1 versus ..._2 -- depends on a choice made at job creation, which the import-time declaration cannot know. Three shapes were tried and committed in turn:

1. Bind the two default monitors and raise at job creation if the operator picked others. That makes the dropdown a lie in wavelength mode.
2. Bind all four candidates, each to its own key, and let the factory (which does see the selection) map the chosen two onto the incident/transmission roles. The two unselected keys become parameters the workflow stores and never computes from, and the candidate list ends up restated in the factory.
3. Let a binding's stream name carry a placeholder -- wavelength_lut/{incident_monitor} -- rendered against the job's actual selection when the gate is resolved. This works, but the name is now declared-but-unrendered in one place and rendered in another, and the Kafka subscriptions (derived at startup, before any job exists) have to be expanded over every declared choice so they stay a superset of whatever a job might render. It was the most intricate mechanism in the PR.

One shared monitor table removes the question instead of answering it: there is a single stream, one provider generic in MonitorType that sciline instantiates per role, and each role picks its rows by the flight path its own graph already computes. Which monitor fills a role is settled by geometry the job already holds, not by the name of the stream it reads. Stream names go back to being fixed at declaration time, which is what the statically derived subscriptions want anyway.

Plotting. With one table per component, the tables appear as N outputs of the LUT workflow. The plot wizard lets you multi-select sources in one step, but pick only one output (a radio button in step 1) -- so comparing components meant adding one layer per table, three wizard steps each, and LOKI's output dropdown was fourteen entries deep with an arbitrary one auto-selected. That asymmetry isn't an oversight: every source of a workflow shares the same output template, so sources are homogeneous by construction and safe to overlay, whereas outputs are different quantities and nothing guarantees they combine. Making outputs multi-selectable would mean building that missing check (intersecting compatible plotters and windowing options, deciding what a mixed-output table column means, and fixing legend labelling, which today counts outputs per cell and would silently collapse every curve to one label). So I built the other fix instead -- #1249, where a result carries the component it is about, putting the tables back on the source axis. It worked; I closed it because the block layout made it unnecessary.

Bookkeeping. The workflow's outputs model had to be generated per instrument with pydantic.create_model (the last such case in the codebase), component names doubled as output field names, and every component cost a stream, a binding and a sciline key.

Underneath all three: a lookup table is a function of distance and event_time_offset alone and carries no component identity, so a per-component table is only a restriction of the same function. The split bought nothing in content -- which is what made it worth paying the block-selection cost to remove. ADR 0011's alternatives table has the short version.

[nvaytet]

I'm still slightly confused: are we publishing only one table or two tables? Which one was it you were showing us screenshots of?

[SimonHeybrock]

Which one was it you were showing us screenshots of?

This morning? That was the released/old version which caries one big table covering the whole range (as configured). On this branch there are two tables, one for detectors (typically small range, but densely covered range), one for monitors (sparse, with some quirks around padding bins etc. that you probably saw).

[nvaytet]

The table for monitors can span many meters (some monitors are close to the source, others close to the sample).
Is it really worth splitting away the table for the detectors? Can't we have one single table for everything with dense regions around the components and sparse regions in-between?

In addition, we made the lookup in numba fast by enforcing a regular grid. If I understood correctly, you find a regular chunk in the table, and then use the number interpolator on that chunk.
How fast/slow is the part that searches for the correct chunk?

[SimonHeybrock]

Good questions.

On splitting. Dense regions with nothing in between is what both tables already are — the gaps carry no rows at all, not even sparse ones. That part is forced by size: BIFROST's monitors span 0.5–155 m, which as one uniform grid at the default 0.1 m resolution is 1549 rows × 286 event_time_offset bins = 3.4 MB to carry 8 rows of information.

So the only open question is whether the detector blocks and the monitor blocks ride in one message or two. Measured from the geometry artifacts at default resolution (2.2 kB/row):

	detector block	monitor blocks
DREAM	77.5–79.5 m, 20 rows, 45 kB	2 monitors, 4 rows, 9 kB
LOKI	24.8–43.9 m, 191 rows, 427 kB	4 monitors, 10 rows, 22 kB
TBL	30.8–32.1 m, 13 rows, 29 kB	1 monitor, 2 rows, 4 kB
BIFROST	(no placeable detector yet)	4 monitors over 0.5–155 m, 8 rows, 18 kB

Two reasons to keep them apart. The detector block is the only large one, and the one that grows with the resolution. Merging hands it to every consumer that wants a monitor: a monitor view job, and both monitor roles of an I(Q) job, each receiving it on every chopper change to read two or three rows.

The other is plotting, which is the more visible one day to day. The two tables need different plotters: the detector table is a single dense uniform block, so it renders as an image; the monitor table is a handful of rows at each monitor with tens of metres of nothing between, so its distance axis is not uniform and it is readable only as "Overlay 1D", one curve per row. Merged, no plotter setting shows both — as an image LOKI's ten monitor rows are invisible slivers on a 37 m axis, as an overlay it is 201 curves. Two outputs means picking one in the wizard and getting a readable picture; one output means building a block selector into the plotting layer to get the same thing back.

Those are the defaults, which are also essreduce's default_parameters() and what DREAM's shipped tables actually use (754 × 287 at 0.1 m / 250 µs). The argument doesn't depend on them: payload scales as 1/(distance_resolution × time_resolution), and the two axes work in our favour — the instrument with the long detector span is the one needing the least resolution (LOKI, 19 m, SANS), while the one that would want a finer grid has a 2 m span (DREAM is still 445 kB at ten times the distance resolution). Finer resolution makes merging worse, never better.

The split itself costs one output field and one stream name; the block machinery is identical either way. I've replaced the ADR's "megabyte-scale" with the measured numbers and the plotting reason.

On the chunk search. It is not in the lookup path. Block selection is a provider taking the context key, so StreamProcessor runs it once per set_context — once per arriving table, i.e. per chopper change — and caches the result for every chunk after that. What reaches WavelengthInterpolator is one uniform block, exactly as a file-based table was; the regular-grid assumption is untouched. That is precisely why select_block exists: handing it the concatenation would read wrong rows silently under numba (and correctly under the scipy fallback).

Cost: 58 µs for LOKI's detector table (one block), 112 µs for BIFROST's monitors (four), 381 µs for an invented 20-block/2000-row table. Most of that is the nanmin/nanmax over the job's Ltotal rather than the block scan — 1.1 ms if Ltotal has 1.5M pixels. For scale, essreduce builds a fresh WavelengthInterpolator on every chunk (44 µs for LOKI's block), so the once-per-table selection is worth about one chunk's interpolator construction.

[nvaytet]

I don't have any more questions.

I did not look at the tests, nor did I check all the code (it was too much to process).
I tried to think about the more general implications.

[SimonHeybrock]

I tried to think about the more general implications.

Thanks, that was exactly what I was after.