feat: add vendor-neutral streaming tool loop - #1826
Conversation
There was a problem hiding this comment.
@arnabnandy7 - thank you for contributing - very valuable! Please see comments. Also, it would be good to have a LangChain4j streaming test similar to the one in the "experimental" repo shared earlier. Thanks again!
Thanks. I’ll add a LangChain4j streaming integration test modeled on LangChainToolLoopIT. |
|
@arnabnandy7 - great development, thank you very much for contributing! |
igordayen
left a comment
igordayen
left a comment
There was a problem hiding this comment.
@arnabnandy7 - added few more comments. praised development.
Thanks I'll take up the next task, but since weekend is over it'll need sometime to clearing up those suggestions. Maybe in next 2-3 days I'll push it. Appreciating your kind words and review. |
|
@arnabnandy7 - could you please verify, time permitting:
|
Verified based on your suggestion:
|
|
@arnabnandy7 - please also keep an eye on PR #1819 file embabel-agent-api/src/main/kotlin/com/embabel/agent/spi/support/streaming/ThinkingStreamSupport.kt |
@igordayen I just checked, yes it'll require conflict resolution in that case I'd request to keep this PR in second in queue so that once #1819 merged I can resolve the conflict in meaningful manner. |
Thanks, @arnabnandy7. Could you please review #1819, please comment on it, especially on the newly introduced Streaming support artifact. Thank you |
|
@igordayen local test results ended up 1 test failure most likely due to low end system configuration and low storage space remaining. Resulted Environment that was used in this test; I conclude with this my machine being too constrained for the tests’ 10/150-second limits. I guess I would require you to run this for all 3 to do a validation. Report.xml A/B comparison confirms these observed failures are not introduced by this PR. Both branches produced the same result:
|
|
@igordayen please proceed with testing, it's passing now; The reduced-load Ollama regression passed completely. Test configuration:
Result: All three paths passed:
Total test-class time was 57.40 seconds, compared with the 8.2B model exceeding the 150-second limits. |
|
Thanks, I will. |
@igordayen If you try with
The reasoning generated around the tool-call boundary is not exposed as separate streaming events. In particular, thinking before/between tool calls is absent from the consumer-visible stream, even though the tool executes and the final response completes successfully. Therefore, the existing Ollama tests confirm that streaming and tooling work, but they do not confirm preservation of thinking across tool calls. This matches the limitation discussed in the issue and is the reason for keeping thinking separate from ordinary content in the vendor-neutral streaming abstraction. PS: I just added LangChain4j streaming integration test as well. |
igordayen
left a comment
igordayen
left a comment
There was a problem hiding this comment.
@arnabnandy7 - good progress, thanks, could you please review comments
|
Tested OpenAI and Anthropic Integration tests: git status Could you please review the logs? |
You are correct that application-level Thinking events appear both before and after tool execution in these logs. The observed sequence is thinking events, two tool calls, another thinking event, streamed objects, and a final thinking event. One distinction is that this test classifies non-JSON streamed prose as Thinking. Therefore, it confirms that application-level thinking events survive the tool boundary, but it does not by itself establish that Anthropic native extended-reasoning blocks are being preserved. Dedicated native-reasoning event coverage would be needed to confirm that separately. |
==> Is this something you manage to have - specifically Anthropic thinking preservation before/after tool call? As I do not remember seeing it before, that was one of the reasons to have this PR. So assume I need to come back to the 'main' branch and re-test, thanks |
@igordayen this PR preserves streamed text across the tool boundary. Content received before the tool call is forwarded, the tool executes, and content from the continuation inference is forwarded afterward. However, after tracing Spring AI’s Anthropic representation, I need to qualify the native-thinking aspect. Spring AI marks Anthropic thinking chunks with Therefore, an A/B run against Would you like me to add the provider-neutral thinking marker and the Anthropic integration coverage to this PR? |
|
Re-tested main branch: it appears that thinking before tool invocation still gets preserved. |
|
Could you please reconfirm unfolding tool behavior. Thank you. |
I'll check and share by tomorrow EOD |
Compiling TODOs:
Thanks. |
Thanks for confirming that thinking before tool invocation remains preserved. I also reconfirmed unfolding-tool behavior using the real LangChain4j/Ollama streaming path. The first request exposes only the unfolding facade. After invoking it, the next request exposes the unfolded child tool, which Ollama successfully invokes before returning its result in the final streamed response. The test passed: This confirms that the unfolded child is added to the actual tool set supplied on the following inference turn, rather than merely being advertised in the facade’s result message. |
905c2e6 to
390032e
Compare
|
@arnabnandy7 - could you please advise on inquiries:
|
I'm working on these To-dos only, thanks for reminding me. 😊 |
390032e to
e1c23f1
Compare
|
@igordayen thanks for consolidating the TODOs. Here is the current status:
As noted, you will run this using:
I restored the original three-argument contract: This preserves compatibility for existing callers and implementations. The new
This now follows the naming of the blocking
The existing public The event-oriented behavior required by the Embabel-managed streaming loop is exposed through the additional default A regression test verifies that the original raw-content streaming signature still works.
The provider-neutral streaming package has no Spring AI imports or Spring AI types in its public contracts. Spring-specific message conversion, aggregation, and model interaction remain under the existing The LangChain4j dependencies are test-scoped and confined to the Ollama autoconfigure module. Verification after these changes: The Ollama integration also reconfirmed that the unfolding facade is removed, the child tool is added to the next inference, and the child is successfully invoked. |
|
Thanks, im offline now, will resume late night. Long drive. Please share your discord contact. Thanks |
I'm available in discord with |
e1c23f1 to
8c9a698
Compare
|
Sanity check is completed and PR description updated including the latest two merge related refactors. It's open for final review. |
igordayen
left a comment
igordayen
left a comment
There was a problem hiding this comment.
@arnabnandy7 - could you please review few inquiries, thank you
Signed-off-by: Arnab Nandy <arnab_nandy7@yahoo.com>
8c9a698 to
3ac9b73
Compare
|
@igordayen latest test results with comments being addressed
|
igordayen
left a comment
igordayen
left a comment
There was a problem hiding this comment.
@arnabnandy7 - long journey, thank you, looks good.
|
Results without native thinking - looks good so far: |
|
With NATIVE Thinking - requires attention: 16:29:16.399 [main] DEBUG CacheEligibilityResolver - Caching not enabled for tool definition, cacheStrategy=NONE [16:29:25.331 [boundedElastic-1] TRACE StreamingLlmOperationsImpl - RAW CHUNK: '<think' LlmOptions thinkingOptions = new LlmOptions().withThinking(Thinking.withTokenBudget(8000)); |
2 participants
Summary
Introduces a vendor-neutral streaming tool loop that mirrors the existing blocking
ToolLooparchitecture while retaining compatibility with the originalLlmMessageStreamercontract.Closes #1800.
Changes
StreamingToolLoopandDefaultStreamingToolLoop.StreamingToolLoop.execute(...)for symmetry with the blockingToolLoop.execute(...)API.LlmMessageStreamer.stream(...): Flux<String>contract.LlmMessageStreamer.streamInference(...)for a single provider-neutral streaming inference.LlmInferenceStreamEventfor incremental content and assembled terminal assistant responses.ChatModel.stream()directly.MessageAggregatorto assemble partial tool-call fragments.ToolInjectionStrategybetween inference turns.UnfoldingToolchildren to become available in subsequent streaming calls.returnDirect.Architecture
The streaming path now follows the same provider-neutral pattern as blocking calls:
Provider integrations perform one streaming inference and expose the assembled assistant response. Embabel manages tool execution, conversation history, dynamic tool injection, and continuation across inference turns.
The streaming loop uses the shared tool-execution support from #1837, keeping inspector callbacks, result conversion, decoration, injection, and deduplication consistent with the blocking loop.
Compatibility
The original public signature remains the functional-interface method:
This preserves the existing method descriptor for callers and implementations.
The new method:
provides the terminal assistant message required by the Embabel-managed streaming tool loop.
Existing implementations can continue returning raw content through
stream(...). Provider adapters that participate in the new loop overridestreamInference(...)to emit incrementalContentevents followed by oneCompleteevent containing the assembled assistant message and any requested tool calls.Tool-call inspectors are owned by
DefaultStreamingToolLoopwhen using the new path because that layer performs the actual tool execution. This provides consistent callback behavior across Spring AI, LangChain4j, and future provider integrations while avoiding provider-specific or duplicate callbacks.Dependency Boundary
The provider-neutral streaming contracts and tool loop contain no Spring AI imports or Spring AI types.
Spring-specific message conversion, response aggregation, and model interaction remain under the existing
spi.support.springai.streamingimplementation package.LangChain4j dependencies are test-scoped and confined to the Ollama autoconfigure module.
Unfolding Tools
The integration coverage verifies the complete unfolding sequence:
This confirms that the child tool is registered for the next inference rather than only being advertised in the facade’s result message.
Validation
Focused streaming unit tests:
Result:
LangChain4j/Ollama integration:
Result: