Skip to content

fix(service): Do not record a model stream that fails after HTTP 200 - #71

Open
sentry-junior[bot] wants to merge 1 commit into
mainfrom
fix/skip-failed-streams
Open

sentry-junior[bot] wants to merge 1 commit into
mainfrom
fix/skip-failed-streams

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

A provider or gateway can send HTTP 200 and then end the stream with an error. The recorder skipped only 429 and 5xx responses, so it stored such a stream, and every later run replayed the failure. getsentry/junior hit this: a Junior eval replayed a 200 stream that ended in AI provider error: timeout on every run, including all of the client's retries.

The recorder now treats a failed event stream like a 5xx. The client still gets the stream as it came, but no recording is written. isFailedStream (streams.ts) reads the stream's own signals, not its error text. A stream failed when it has:

  • an error event: event: error, a type of error or response.failed, or an error object (OpenAI Chat Completions)
  • a finish reason of error (finishReason, finish_reason, or stop_reason)
  • no end: an Anthropic Messages stream with message_start but no message_stop, or an OpenAI Chat Completions stream with no finish reason

These are the same checks that the Pi Anthropic client uses when it decides a stream failed (event: error, or "stream ended before message_stop"). A stream that drops its connection mid-body was already safe: readBody rejects, so nothing is recorded.

Streams in other formats are recorded as before unless they have one of the error signals above. The stream in deployed.test.ts gains the message_stop that a real Anthropic stream always sends.

via @BYK.

--

View Session [Sentry]

A provider or gateway can send HTTP 200 and then end the stream with an
error, such as a gateway timeout. The recorder skipped only 429 and 5xx
responses, so it kept such a stream, and each later run replayed the
failure.

The recorder now treats a failed event stream like a 5xx: the client
gets the stream, but no recording is written. A stream failed when it
has an error event, a finish reason of `error`, or no end (an Anthropic
Messages stream without `message_stop`, or an OpenAI Chat Completions
stream without a finish reason).

Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
@sentry-junior
sentry-junior Bot requested a review from BYK November 16, 2026 20:49
@BYK
BYK requested review from dcramer and removed request for BYK November 16, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant