Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions articles/flow/ai-support/controllers.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@ The built-in controllers cover grid and chart data exploration. To expose your o
[classname]`AIController` defines three methods:

* [methodname]`getTools()` -- returns the list of [classname]`LLMProvider.ToolSpec` instances the controller contributes to each LLM request. Tools are collected before every request, so a controller can vary its tool set based on current state.
* [methodname]`onRequest()` -- runs on the UI thread right before the LLM stream opens. Use it to lock UI surfaces, snapshot state the tool definitions depend on, or otherwise prepare for the turn.
* [methodname]`onResponse(Throwable)` -- runs on the UI thread once the LLM stream has completed, either successfully (`error` is `null`) or with an error. Controllers use this hook to commit deferred state changes on success and release any per-turn state captured in [methodname]`onRequest()` on failure.
* [methodname]`onRequest()` -- runs on the UI thread right before the LLM stream opens. Use it to lock UI surfaces, snapshot state the tool definitions depend on, or otherwise prepare for the turn. Since tools may execute on a background thread, this is also the place to capture anything that depends on Vaadin thread locals, such as [methodname]`UI.getCurrent()`.
* [methodname]`onResponse(Throwable)` -- called through `ui.access()` once the LLM stream has completed, either successfully (`error` is `null`) or with an error, so it can safely update components. Controllers use this hook to commit deferred state changes on success and release any per-turn state captured in [methodname]`onRequest()` on failure.

Each tool is an implementation of [classname]`LLMProvider.ToolSpec`, which has four methods:

Expand Down
4 changes: 3 additions & 1 deletion articles/flow/ai-support/conversation-history.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ The listener fires once per turn for both successful and failed exchanges -- che

.UI Updates from ResponseListener
[IMPORTANT]
The listener runs on a background thread. It is safe to perform blocking I/O (such as database writes) directly. However, to update Vaadin UI components from this callback, wrap the update in `ui.access()`.
With a streaming provider, or when <<llm-providers#background-execution,background execution>> is enabled, the listener runs on a background thread, where blocking I/O (such as database writes) is safe. With a synchronous provider in the default execution mode, the whole turn -- this listener included -- runs in the request that triggered the prompt, so blocking work extends that request. To update Vaadin UI components from this callback, wrap the update in `ui.access()`.


== Restoring History
Expand Down Expand Up @@ -96,6 +96,8 @@ orchestrator.reconnect(provider)
.apply();
----

Provider settings such as the streaming mode and <<llm-providers#background-execution,background execution>> live on the provider and are not serialized; apply them to the new provider instance before passing it to [methodname]`reconnect()`.

.Controller Reattachment
[NOTE]
[classname]`AIController` instances, including [classname]`GridAIController`, [classname]`ChartAIController`, and [classname]`FormAIController`, are not serialized with the orchestrator. Create a new controller after session restore and pass it to [methodname]`withController()` on the reconnector. The grid and chart controllers each have their own state capture and restoration API. The form controller has no separate state object, because the form fields themselves are persisted with the [classname]`VaadinSession`. See <<ai-powered-grid#,AI-Powered Grid>>, <<ai-powered-chart#,AI-Powered Chart>>, and <<ai-powered-form#,AI Form Filler>> for the specifics.
Expand Down
3 changes: 2 additions & 1 deletion articles/flow/ai-support/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ A system prompt is strongly recommended. Without one, the LLM has no guidance be
Each [classname]`LLMProvider`, [classname]`MessageList` (or [classname]`AIMessageList`), [classname]`MessageInput` (or [classname]`AIInput`), file receiver, and [classname]`AIController` may be passed to only one [classname]`AIOrchestrator`. Attempting to share an instance across two orchestrators throws [classname]`IllegalStateException` at build time. When building multi-view or dashboard applications, create a separate set of components -- including a dedicated provider -- for each orchestrator.


[[server-push]]
== Server Push

Streaming mode pushes partial responses to the UI as tokens arrive. This requires server push to be enabled. Annotate your application shell with [annotationname]`@Push`:
Expand All @@ -86,7 +87,7 @@ public class Application implements AppShellConfigurator {
----

[TIP]
Synchronous mode does not require push. If push is not available in your environment, disable streaming on the provider. A warning is logged at runtime if push is not enabled when using streaming mode.
Synchronous mode does not require push. If push is not available in your environment, disable streaming on the provider. A warning is logged at runtime when neither automatic push nor polling is active while using streaming mode. Note that a synchronous provider runs the whole exchange in the request that triggered it, which blocks the UI until the response is complete -- see <<llm-providers#background-execution,Background Execution>> for keeping the UI responsive.


== Topics
Expand Down
34 changes: 34 additions & 0 deletions articles/flow/ai-support/llm-providers.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ Streaming is enabled by default. To disable it, call [methodname]`setStreaming(f
provider.setStreaming(false);
----

In synchronous mode, the whole exchange runs in the request that triggered it and blocks the UI until the response is complete. See <<#background-execution,Background Execution>> for keeping the UI responsive during long prompts.

.History Restoration with ChatClient
[NOTE]
History restoration via [methodname]`withHistory()` is only supported when creating the provider from a [classname]`ChatModel`. Providers created from a [classname]`ChatClient` do not provide access to internal memory, so calling [methodname]`setHistory()` throws an [classname]`UnsupportedOperationException`. Use `new SpringAILLMProvider(chatModel)` if you need to restore conversation history across sessions.
Expand All @@ -67,6 +69,36 @@ LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel);

The provider manages its own conversation memory using a 30-message window.

Synchronous mode blocks the UI for the duration of each exchange; see <<#background-execution,Background Execution>>.


[[background-execution]]
[role="since:com.vaadin:vaadin@V25.3"]
== Background Execution

A synchronous provider produces the response on the thread that asks for it. For a prompt sent from the browser, that is the request thread: the request does not return until the model has produced the complete response, including any tool calls along the way. The interface freezes for the whole wait, and since the request holds the session lock, other views in the same session wait too.

Background execution moves the exchange to a background thread instead. Enable it on either built-in provider:

[source,java]
----
provider.setBackgroundExecution(true);
----

The user's message and an empty assistant message then appear immediately, the UI stays responsive, and the response is filled in when the model finishes. The setting is off by default. It's read for each prompt, so it can be changed at any time; the next prompt uses the new mode. It has no effect with a streaming model, whose response already arrives on the LLM client's own threads.

The response is now produced outside any request, so it reaches the browser through server push or polling. Enable push by annotating the application shell with [annotationname]`@Push` (see <<index#server-push,Server Push>>), or enable polling with [methodname]`UI.setPollInterval()`. Without either, the response only shows up with the next request the browser happens to make -- the page looks stuck even though the turn completed on the server. The provider logs a warning, once per provider instance, when neither is active. Manual push mode is not enough on its own, because nothing in the framework calls `ui.push()` for the application.

Everything that happens before the model is called still runs in the request thread: the request interceptor, adding the user's message and the empty assistant message to the Message List, [methodname]`AIController.onRequest()`, the request listener, and the session context supplier. The model calls, every tool execution, and the [classname]`ResponseListener` run on the background thread, where [methodname]`UI.getCurrent()` and other Vaadin thread locals return `null` and components must not be touched directly. Thread-bound framework state, such as Spring Security's [classname]`SecurityContext`, is absent there for the same reason.

Wrap component access in `ui.access()`, or capture what a tool needs in [methodname]`AIController.onRequest()` while the request thread is still current -- see <<tool-calling#,Tool Calling & Programmatic Prompts>> and <<controllers#,Controllers>>. The built-in controllers already handle this. [methodname]`AIController.onResponse()` is the exception: the orchestrator calls it through `ui.access()`, so it can update components directly.

.One Prompt at a Time
[NOTE]
The orchestrator processes one prompt at a time. In the default synchronous mode, a message submitted while a turn is running waits for the session lock and is processed when the turn ends. With background execution the lock is free, so the same message is rejected and dropped with a server-side warning -- and the Message Input has already cleared its text.

If the user closes or reloads the browser tab while a turn is running, the turn still completes on the server: the response is recorded in the conversation history and the [classname]`ResponseListener` fires as usual. Only the UI updates are skipped, along with [methodname]`AIController.onResponse()`, which needs an attached UI. The setting itself lives on the provider and is not serialized with the session -- apply it again to the recreated provider after a session restore, before passing it to [methodname]`reconnect()`. See <<conversation-history#,Conversation History & Session Persistence>>.


== Custom LLM Providers

Expand All @@ -93,4 +125,6 @@ public class MyLLMProvider implements LLMProvider {
}
----

The orchestrator calls [methodname]`stream()` on the thread that triggers the prompt and subscribes to the returned stream on that same thread -- whether a turn runs in the background is decided entirely by the implementation. An implementation whose LLM call blocks should schedule that call itself; otherwise it occupies the request thread and holds the session lock for the whole turn. See <<#background-execution,Background Execution>> for how the built-in providers expose this as a setting.

endif::flow[]
4 changes: 4 additions & 0 deletions articles/flow/ai-support/tool-calling.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ var orchestrator = AIOrchestrator

For Spring AI, use [annotationname]`@org.springframework.ai.tool.annotation.Tool`. For LangChain4j, use [annotationname]`@dev.langchain4j.agent.tool.Tool`.

.Tool Threading
[NOTE]
With a streaming provider or <<llm-providers#background-execution,background execution>>, tool methods are invoked off the request thread, where [methodname]`UI.getCurrent()` and other Vaadin thread locals are not available. Wrap component access in `ui.access()`, or capture the needed state before the turn starts.

.Framework-Agnostic Tools via Controllers
[TIP]
For a reusable set of tools that does not depend on a specific LLM framework's annotations, or when a lifecycle hook is needed after each LLM request cycle, implement <<controllers#,[classname]`AIController`>> instead. [classname]`GridAIController` and [classname]`ChartAIController` are built-in examples. Controllers and tool objects can be combined on the same orchestrator.
Expand Down
Loading