Deliver traffic whose timestamp has stopped advancing - #1284
Merged
Conversation
A producer with a stuck clock never converges a rate estimate and never moves the high-water mark past the timeout threshold, so as the only traffic in a service nothing ever closes the window. The stall backstop saw the anchor inside the window, took it for quietness and re-armed, and the buffer grew at the ingest rate with nothing delivered. Held-back messages in _future were invisible to the stall check altogether. Quietness and a stuck clock differ in one observable: whether the buffer has grown since the previous stall check. When it has, the backstop fires the timeout close, which delivers the traffic and advances the data clock by one batch length per stall interval -- the one case where wall time moves the data clock, because the data clock itself has stopped. The stall check now counts held-back traffic as buffered. Symptom of #1275. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SimonHeybrock
force-pushed
the
1283-stuck-clock-stall-close
branch
from
September 3, 2026 13:39
15070e7 to
542d442
Compare
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A producer whose timestamp stops advancing can never satisfy a slot gate (zero diffs are filtered, so no rate converges) and never moves the high-water mark past the timeout threshold. If it is the only traffic in a service, nothing ever closes the window. The wall-clock stall backstop saw the anchor inside the window, took the stall for quietness and re-armed, so the buffer grew at the full ingest rate while nothing was delivered. Held-back messages in the future queue were invisible to the stall check altogether, which is the variant the review of #1271 found.
Symptom of #1275, which tracks the underlying missing bound on active-window residency. This PR bounds only the stuck-clock case.
What this does
Quietness (a stopped stream's trailing partial batch) and a stuck clock look identical at a single stall check and differ in one observable: whether the buffer has grown since the previous check. When it has, the backstop fires the timeout close, which delivers the traffic and advances the data clock by one batch length per stall interval. That is the one case where wall time moves the data clock, and it does so because the data clock itself has stopped. The buffer is bounded at a couple of stall intervals' worth of traffic, and every message is delivered.
The stall check now counts held-back traffic as buffered, so a future-queue-only stall recovers like a bucket stall.
The trailing-partial-batch case is unchanged: a buffer that does not grow still re-arms silently, and the existing no-thrash test pins that.
Test plan
test_stuck_clock_is_delivered_and_bounded, parametrized over the bucket and future-queue variants: everything fed is delivered or still held, and the held depth stays under three stall intervalstests/core tests/services tests/kafka: 971 passed, 1 skipped, 4 xfailed (pre-existing)