Skip to content

Deliver traffic whose timestamp has stopped advancing - #1284

Merged
MridulS merged 3 commits into
mainfrom
1283-stuck-clock-stall-close
Sep 7, 2026
Merged

Deliver traffic whose timestamp has stopped advancing#1284
MridulS merged 3 commits into
mainfrom
1283-stuck-clock-stall-close

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Sep 3, 2026

Copy link
Copy Markdown
Member

Motivation

A producer whose timestamp stops advancing can never satisfy a slot gate (zero diffs are filtered, so no rate converges) and never moves the high-water mark past the timeout threshold. If it is the only traffic in a service, nothing ever closes the window. The wall-clock stall backstop saw the anchor inside the window, took the stall for quietness and re-armed, so the buffer grew at the full ingest rate while nothing was delivered. Held-back messages in the future queue were invisible to the stall check altogether, which is the variant the review of #1271 found.

Symptom of #1275, which tracks the underlying missing bound on active-window residency. This PR bounds only the stuck-clock case.

What this does

Quietness (a stopped stream's trailing partial batch) and a stuck clock look identical at a single stall check and differ in one observable: whether the buffer has grown since the previous check. When it has, the backstop fires the timeout close, which delivers the traffic and advances the data clock by one batch length per stall interval. That is the one case where wall time moves the data clock, and it does so because the data clock itself has stopped. The buffer is bounded at a couple of stall intervals' worth of traffic, and every message is delivered.

The stall check now counts held-back traffic as buffered, so a future-queue-only stall recovers like a bucket stall.

The trailing-partial-batch case is unchanged: a buffer that does not grow still re-arms silently, and the existing no-thrash test pins that.

Test plan

  • test_stuck_clock_is_delivered_and_bounded, parametrized over the bucket and future-queue variants: everything fed is delivered or still held, and the held depth stays under three stall intervals
  • tests/core tests/services tests/kafka: 971 passed, 1 skipped, 4 xfailed (pre-existing)

@SimonHeybrock SimonHeybrock added the area:backend Services, Kafka, message pipeline, preprocessors, job handling label Sep 3, 2026
A producer with a stuck clock never converges a rate estimate and never
moves the high-water mark past the timeout threshold, so as the only
traffic in a service nothing ever closes the window.  The stall backstop
saw the anchor inside the window, took it for quietness and re-armed,
and the buffer grew at the ingest rate with nothing delivered.  Held-back
messages in _future were invisible to the stall check altogether.

Quietness and a stuck clock differ in one observable: whether the buffer
has grown since the previous stall check.  When it has, the backstop
fires the timeout close, which delivers the traffic and advances the
data clock by one batch length per stall interval -- the one case where
wall time moves the data clock, because the data clock itself has
stopped.  The stall check now counts held-back traffic as buffered.

Symptom of #1275.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MridulS
MridulS merged commit 48a567f into main Sep 7, 2026
19 checks passed
@MridulS
MridulS deleted the 1283-stuck-clock-stall-close branch September 7, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Services, Kafka, message pipeline, preprocessors, job handling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants