Support streaming LLM response

[mekkiamiri]

Currently, the prompt runner call is blocking:
For example: String response = this.ai.withLlm(options).generateText(prompt);
So for long time running prompts, call fails with timeout.
It is very helpful to stream response like python frameworks(crewAI for example). If we can have something like:
Flux<String> response = this.ai.withLlm(options).streamText(prompt);

[alexheifetz]

looping @johnsonr

[LiveNathan]

I was coming here to ask the same thing. I guess Embabel doesn't support streaming token responses because its focus is not not chat?

[igordayen]

please refer to the recent post on Embabel discord, "general" chat:

LLM Streaming Support is available now, in agent snapshot build 0.3.1-SNAPSHOT. Please refer to User Guide on usage: embabel-agent/embabel-agent-docs/src/main/asciidoc/reference/streaming/page.adoc.
Java examples:
embabel-agent/embabel-agent-autoconfigure/models/embabel-agent-openai-autoconfigure/src/test/java/com/embabel/agent/config/models/openai/LLMOpenAiStreamingBuilderIT.java,
embabel-agent/embabel-agent-autoconfigure/models/embabel-agent-anthropic-autoconfigure/src/test/kotlin/com/embabel/agent/config/models/anthropic/LLMAnthropicStreamingBuilderIT.java,
embabel-agent/embabel-agent-autoconfigure/models/embabel-agent-ollama-autoconfigure/src/test/java/com/embabel/agent/config/models/ollama/LLMOllamaStreamingBuilderIT.java
If you observe any issues - especially for models besides tested above, - please raise your hands

[igordayen]

Closing as implemented, @mekkiamiri @LiveNathan - FYI