Fallback to sync invocation if streaming fails

[hubertzub-db]

Summary

• When the upstream streamText call encounters an error mid-stream, the chat server now transparently falls back to a non-streaming generateText call so the user still gets a response instead of just an error message.
• I had to replace the previous writer.merge() approach with a manual drainStreamToWriter helper that reads chunks individually, detects errors, and triggers the fallback path when needed.

Demo/test (simulated error in agent's stream_handler)

stream-fallback.mov

Regression test (streaming still works)

stream-regression.mov

Demo of complete failure (both stream_handler and invoke_handler in agent are failing)

stream-complete-failure.mov

Test plan

• Verify normal streaming still works end-to-end (no regression)
• Simulate a streaming failure (e.g. upstream timeout or model error) and confirm the fallback fires and the user receives a complete response
• Confirm error logging appears in the server console when fallback is triggered
• Verify that if both streaming and fallback fail, the client receives a data-error message

[hubertzub-db]

note: unfortunately we have to merge the streams manually, otherwise we won't have enough control to transparently switch over to fallback :/

[bbqiu]

ah sorry i think i wasn't clear, but we only want to try the non-streaming path if the streaming path never starts -- for errors that are within the stream itself, we do not want to falback right? this could result in duplicate content

[bbqiu]

ex. when there is a network disconnect, we shouldn't be retrying the synchronous path, we should be trying to resume the stream.

pr looking at how the stream resumption works for context: #121

we could maybe? look at whether or not any content was emitted (ex. there was never a SSE w/ content in it?)

[hubertzub-db]

ah sorry i think i wasn't clear, but we only want to try the non-streaming path if the streaming path never starts

If the streaming never starts because agent's streaming handler crashes at the beginning, then the chunk.value.type in middleware's drainStreamToWriter loop will have be set to error. That's due to agent framework/aisdk detecting a chunked stream even if the agent's stream function was never registered. In other words, even if agent<->middleware streaming completely fails at the very start, then the stream is still detected and passes initial messages (handshakes etc).

[hubertzub-db]

hmm since that's not trivial, maybe let's discuss this offline

[bbqiu]

thank you for working on this! i wasn't clear in specifying what type of fallback we wanted, but if it's possible to only fallback when the stream doesn't start in the first place, that would be very helpful

also, once we make the change, if we could write a playwright integration test for this, that would be super helpful

[hubertzub-db]

@bbqiu I've

• updated the logic to wait until first meaningful text chunk
• adjusted so it works with transporting traceIds (Add thumbs up/down feedback to chatbot app #94)
• added tests