Skip to content

fix: Keep requested keys alive in the naive scheduler - #240

Merged
SimonHeybrock merged 2 commits into
mainfrom
naive-scheduler-keep-requested-keys
Sep 2, 2026
Merged

fix: Keep requested keys alive in the naive scheduler#240
SimonHeybrock merged 2 commits into
mainfrom
naive-scheduler-keep-requested-keys

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

NaiveScheduler frees an intermediate result as soon as every provider consuming it has run. The consumer count is built from the graph's providers only, so a key that is both requested and consumed by another task reaches zero and is deleted before it can be returned:

A = NewType('A', int)
B = NewType('B', int)

def make_b(a: A) -> B:
    return B(a + 1)

pl = sciline.Pipeline((make_b,))
pl[A] = 1

pl.compute((A, B))                                    # dask:  {A: 1, B: 2}
pl.get((A, B), scheduler=NaiveScheduler()).compute()  # naive: KeyError(A)

Any compute for several targets where one target depends on another is affected; the dask scheduler is not. The regression came in with a5c9a08 ("Discard data early in naive scheduler") and is in 26.8.0, so 25.11.1 and earlier are fine.

Counting the requested keys as consumers keeps them alive until they are returned, and everything else is still discarded as early as before.

This was found from downstream fallout rather than from reading the code: ESSlivedata defaults its services to the naive scheduler, and there both detector-view construction and monitor workflow cycles fail with a KeyError on a live key. ess.reduce.streaming.StreamProcessor hits the pattern routinely, computing several targets at once where one feeds another. With this fix those workflows run again under the naive scheduler.

Test plan: the new test runs on all three scheduler fixtures and fails only on [naive] without the fix.

NaiveScheduler frees an intermediate result as soon as every provider consuming
it has run. Requested keys were not exempt from this, so a key that is both
requested and consumed by another task was deleted before it could be returned,
raising KeyError.

This affects any compute for multiple targets where one target depends on
another, for example `pipeline.compute((A, B))` with a provider `A -> B`. The
dask scheduler is unaffected.

Excluding the requested keys from the discard step keeps them alive until they
are returned, while everything else is still discarded as early as before.
@SimonHeybrock
SimonHeybrock force-pushed the naive-scheduler-keep-requested-keys branch from 0fc3dcc to e1b3239 Compare August 31, 2026 06:30
@SimonHeybrock
SimonHeybrock requested a review from jl-wynen August 31, 2026 06:39
@SimonHeybrock SimonHeybrock changed the title Keep requested keys alive in the naive scheduler fix: Keep requested keys alive in the naive scheduler Aug 31, 2026
Comment thread src/sciline/scheduler.py Outdated
Comment on lines +88 to +97
requested: Container[Hashable],
) -> None:
"""Discard results that no remaining provider needs.

Requested keys are kept: they are returned to the caller, so their consumer
count reaching zero does not mean they are no longer needed.
"""
for arg in provider.arg_spec.keys():
counts[arg] -= 1
if counts[arg] == 0:
if counts[arg] == 0 and arg not in requested:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
requested: Container[Hashable],
) -> None:
"""Discard results that no remaining provider needs.
Requested keys are kept: they are returned to the caller, so their consumer
count reaching zero does not mean they are no longer needed.
"""
for arg in provider.arg_spec.keys():
counts[arg] -= 1
if counts[arg] == 0:
if counts[arg] == 0 and arg not in requested:
requested: set[Hashable],
) -> None:
"""Discard results that no remaining provider needs.
Requested keys are kept: they are returned to the caller, so their consumer
count reaching zero does not mean they are no longer needed.
"""
for arg in provider.arg_spec.keys() - requested:
counts[arg] -= 1
if counts[arg] == 0:

This is an internal function, so it doesn't need to have a general argument. This way, you avoid repeated lookups.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point on the annotation, changed to set[Hashable].

I kept the loop as it was, though: ArgSpec.keys() is a generator, so keys() - requested raises TypeError -- making it work would need a set(...) construction per call. And the lookup isn't actually repeated: arg not in requested sits behind counts[arg] == 0 and short-circuits, so it runs at most once per provider, for the single argument whose count reaches zero.

_consume_arguments is internal and always called with a set.
@SimonHeybrock
SimonHeybrock merged commit ff6267a into main Sep 2, 2026
6 checks passed
@SimonHeybrock
SimonHeybrock deleted the naive-scheduler-keep-requested-keys branch September 2, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants