Rationalize Thinking in Streaming mode

[igordayen]

@jorander, @Azarux, @johnsonr

Was looking for a PR (if you find one, please let me know) that added logic to Anthropic Options Converter:

override fun convertOptions(options: LlmOptions): AnthropicChatOptions {
val builder = AnthropicChatOptions.builder()
.temperature(options.temperature)
.topP(options.topP)
.maxTokens(options.maxTokens ?: DEFAULT_MAX_TOKENS)
.thinking(
if (options.thinking?.enabled == true) AnthropicApi.ChatCompletionRequest.ThinkingConfig(
AnthropicApi.ThinkingType.ENABLED,
options.thinking!!.tokenBudget,
) else AnthropicApi.ChatCompletionRequest.ThinkingConfig(
AnthropicApi.ThinkingType.DISABLED,
null,
)
)
Here is the problem statement: usability of the streaming API:

LlmOptions thinkingOptions = new LlmOptions().withThinking(Thinking.withTokenBudget(8000));
PromptRunner runner = ai.withDefaultLlm().withLlm(thinkingOptions);
assertTrue(runner.supportsStreaming(), "Default LLM must support streaming");

        Flux<StreamingEvent<ParkingRecommendation>> stream = new StreamingPromptRunnerBuilder(runner)
            .streaming()
            .withPrompt(PARKING_PROMPT)
            .createObjectStreamWithThinking(ParkingRecommendation.class);

As can be seen,

createObjectStreamWithThinking
requires Thinking.withTokenBudget(8000)); as pre-req.

Seems unnatural to me.

Since the converter was moved to another module via just a copy, git history was lost, and I was not able to track PRs.

@jorander, @azanux - appreciate it if you might remember this change.

Options:

• warning on the API if LLMOption does not activate Thinking
• auto-activate in API using a reasonable default if not set

Note: unlike two separate modes, creating() and thinking(), streaming() includes both. One API returns a collection of object streams, while stream+think returns a collection of streaming events.

Thank you.

[jorander]

@igordayen I don't know from what PR the configuration code comes, but I think it is reasonable to argue for the current functionality as well.

Setting thinking budget/level through configuration is a way to tell the LLM you want it to be a bit extra thorough before returning its response. Using *withThinking is a way to retrieve whatever thinking the LLM is doing through the Embabel API. Even though they often are connected I can see reasons for using one without the other:

• Setting thinking budget to push the LLM to reason without being interested in the actual thinking tokens produced.

• Retrieving thinking tokens without setting a budget for a model where thinking is always on and can't be configured on a per call basis.

Adding defaults or warnings might be to make to many assumptions on usage scenarios. Perhaps just making the docs explicit with the current functionality is the best way forward.

[igordayen]

@jorander - thanks for the input.

Would adding a warning be fine then?

[jorander]

@igordayen Having read your article on LLM tool call reasoning via the system prompt and looked at the complete example I think that adds another scenario where you want to call *WithThinking without necessarily having configured a thinking budget. So, I think even a warning in those cases might confuse more than it helps, and add noise to the log.

[igordayen]

@jorander , @azanux , @arimu1 @arnabnandy7

Apologies, but the issue I raised is no longer valid:

As can be seen,

createObjectStreamWithThinking
requires Thinking.withTokenBudget(8000)); as pre-req.

Does not hold any longer.

@jorander suspicion was correct.

Basically, creating an object stream with thinking works perfectly fine without setting a budget.

Budget just adds an extra dimension: "do more rigorous reasoning". It's just an add-on.

Going to close this issue and the respective PR.

Thank you