Rationalise Thinking Prompt Behavior

[igordayen]

OVERVIEW

Let's consider a test

void createObjectWithThinkingResult() {
        logger.info("Running thinking positive test");

        PromptRunner runner = ai.withDefaultLlm(); // <1>
        assertTrue(runner.supportsThinking(), "Expected the prompt runner to support thinking"); // <2>

        ThinkingResponse<TravelPlan> response = runner
            .withSystemPrompt( // <3>
                """
                     You are a travel assistant.
                     You MUST:
                   1. Provide reasoning inside <decision_reasoning>...</decision_reasoning>
                   2. Keep reasoning concise (3-5 bullet points)
                   3. Explain why the itinerary is optimal.

                   <decision_reasoning>
                   Explain:
                   - time constraint
                   - risk of being late
                   - trade-offs
                   </decision_reasoning>
                   """)
            .thinking() // <4>
            .createObject("""
                Create a short travel plan for a three-day trip from London to Paris.
                Include major landmarks.
                Make the most optimal and balanced itinerary.
                """, TravelPlan.class); // <5>

        logger.info("Positive thinking response: {}", response);
        logger.info("Positive thinking content: {}", response.getThinkingContent());

        assertNotNull(response);
        assertTrue(response.hasResult());
        assertNotNull(response.getResult());
        assertTrue(response.hasThinking()); // <5>
    }

Worth noting:

• The prompt specifically asks to generate reasoning with a proper tag
• thinkg() - just indicates expected result ThinkingResponse
• framework will not strip non-JSON within LLM response, unlike non-thinking counterpart, but rather parse it as reasoning

Would like to discuss (and maybe convert issue into Discussion) the following consequences.
What if the user did not include any request for reasoning in the prompt?
There is no enforcement to generate reasoning.

Possibilities:

• syntax```thinkg("some-tag") ````, and auto-add some standardized system message "You MUST generate reasoning with <some-tag"
• create standard ThinkingSystemMessageTransformer - will append to the system message the same
  But what IF a user-created prompt with reasoning includes tag A, but some-tag=tag B?
  Trust that the user is well aware and will use only one of the mechanisms: regular prompting OR auto-generated system prompt?

Note - this is different from another issue on Thinking - to have a uniform solution across non-streaming and streaming and refine native-thinking-support: #1716

@jimador, @azanux - could you please review?

[igordayen]

@jimador, @azanux @jorander - would like to comment that, unlike streaming with a separate prompt that includes thinking instructions. Thinking mode does not have a dedicated prompt. Considering that route would be yet another option. With a fixed tag within.
Seems to work well so far for streaming mode.

[jorander]

@igordayen I think we should avoid mixing LLM internal thinking with application level reasoning or decision motives like "<decision_reasoning>". I think "thinking" in Embabel API should be reserved for the internal LLM thinking.

If, as in this example, an application prompt requests some sort of reasoning or motivation for the decision made by the LLM I think it should be extracted and returned as part of the domain object created, not as thinking in the Embabel API. If the application wants to request a motivation it also has to supply a domain object where it can be returned.

[igordayen]

@jorander @azanux @Abtiotm

Thanks for the input.

The decision to use non-structured reasoning was not easy; multiple iterations with RJ.
See the recent blog on semi-structured reasoning:

https://www.baeldung.com/java-embabel-ai-agent-tool-call-reasoning.

Placeholders can be used for extracting a map from reasoning, as it's just a simple POJO with primitive types.

In addition, we experimented with non-structured output processing using Stanford NLP to analyze reasoning further.

And nothing prevents the user from incorporating structured reasoning into the object itself, of course.

[jorander]

@igordayen Thanks for the link to the article. I hadn't seen it before. Interesting read and I think I see what you are aiming for here. Having the opportunity to instruct the LLM to summarize its reasoning before making decisions/recommendations or tool calls is powerful, both for agent debug/improvement and for exposing to the user to build trust in the agent.

With that said I see a few issues with the current implementation:

Thinking/reasoning are overloaded terms in the context of LLMs

Outside Embabel those words refer to the LLMs native thinking as most often can be configured (thinking budget in Embabel) and is returned separately from the generated content (ref issue #1716).
In Embabel (according to the article) the application level reasoning specifically requested in the prompt is generated by the LLM as part of the content and extracted into ThinkingBlocks by the framework. In the API this type of reasoning/justification is also exposed as "thinking".

I think we need to make the distinction between those two concepts clear in the API so that content can be routed differently in the agent application. Introducing a whole new term and methods (like ".reasoning()" and "*WithReasoning") might be impractical, especially as you might want to combine them with native thinking data. Perhaps some form of differenting property on ThinkingBlock is a more convenient solution?

Dependency on magic strings between prompt and Embabel thinking content extraction

Though it is a good thing the prompt can be used to instruct the LLM on what type of reasoning/justification to return, in doing so tags must be chosen so that the Embabel extraction implementation will find them and extract the enclosed reasoning. The set of valid tags, however quite extensive, is hidden deep in the framework code and only available to an agent developer through the documentation.

To make it fully up to the application code to decide on what tag to use I would suggest that the added .thinking(tag) method in this issue/PR is repurposed from prompt contribution to being the tag used for extracting thinking content. Hence the current hard coded list of strings (tags) could be removed. The methods could even be changed to accept a vararg, facilitating extraction of thinking content from several different tags.

Of course, the same parameter should be added to the *WithThinking methods for symmetry in the streaming API.

I see little value in the suggested PromptContributer solution since an agent developer more likely wants to specify the desired reasoning/justification in the prompt rather than having a default text included. To strengthen the connection between the tag in the metod parameter and the prompt we could add a warning (error?) if the supplied tags cannot be found in the system prompt.

[igordayen]

@jorander - thank you for thinking about thinking:)
@azanux @kenkc ,@Abtiotm - FYI
One of the interesting features of the framework is dynamic tags. No pre-determined ones, full dynamism. Any tag with the name is auto-detected.
Indeed, we witness a trend of shifting from prompt-based reasoning to provider-supported.
I feel that any standardization in this area will take time, and we have to deal with this.
Standard will instruct "provide with reasoning"; however, in depth-reasoning instructions will still be left to the developer (that's why I suggested 3 tests to confirm behavior).
As of now, provider-supported thinking instruction is supported for Gemini.
Separately, Embabel thinking mode got enabled for Anthropic and OpenAI through budgeting.
Was able to track this:

and respective PR:

#1691

That being said, it's theoretically OK to have two levels of thinking, through a system prompt and by low-level provider instruction.
Thinking tags detector should be able to detect both types of reasoning irrespective of origination.

Thanks all for continuing the discussion.

[jorander]

@igordayen I'm sorry, I missed the dynamicTagsDiscoveryAndExtraction function in embabel-common. Hence the TAG_DEFINITIONS map isn't the only tags to be discovered. Any XML-tag returned by the LLM will be extracted as thinking blocks for blocking LLM calls.

For streaming calls I find the implementation in ThinkingDetector in embabel-agent. From just reading the code and unit test I can't see any dynamic discovery there, only tags in the predefined list seems to be detected for extraction.

My conclusions so far:

1. For blocking calls the change proposed in the PR seems reasonable, even though I cannot really see the value in a predefined prompt entry with a dynamic tag name.

2. We need to make sure the implementation for streaming calls catches up functionality wise.

3. At the latest when Make NATIVE thinking-content available in both blocking and streaming calls through ThinkingResponse/StreamingEvent #1716 is implemented we should decide on how to differentiate between the native and application level thinking blocks as per discussed in my previous comment above.

[igordayen]

@jorander - correct. ``thinking()``` API is only for blocking calls as per this issue. Thanks.

#1716 - very important issue; let's resume after completing post 1.0.0 merge activities. The branch is locked as of now.

[azanux]

Thanks @jorander @igordayen @Abtiotm, really useful thread. One angle that might help settle the extraction question:

Beyond ergonomics, dynamic tag discovery has a correctness cost: the regex has no notion of a "reasoning tag" - <([a-zA-Z][a-zA-Z0-9_-])[^>]>(.*?)</...> treats div exactly like think. Asking for an HTML component gives

"<div class=\"card\">Hello</div>{...}" → ThinkingBlock(content=Hello, tagValue=div)

The result object is fine, but hasThinking() is now true and getThinkingContent() returns a content fragment, with no reasoning requested anywhere. The no_prefix rule (^(.*?)(?={)) does the same for any prose before the first {, tag or not. That matters most for the use case @jorander described — surfacing reasoning to the user to build trust. #1788 was the same class of issue one layer down.

So I'd lean toward @jorander 's thinking(vararg tags) as an extraction selector rather than a prompt contributor: it bounds what gets captured. One thing worth keeping from the current PR, though: a warning when a declared tag is absent from the system prompt, so a silent hasThinking() == false becomes a diagnostic.

On native vs application-level thinking (#1716) - no need to decide now, but one constraint to keep in mind: ThinkingBlock's tagType (TAG/PREFIX/NO_PREFIX) discriminates syntax, not origin. Native thinking blocks carry opaque provider state ,Anthropic attaches a signature and Gemini does the same (signature on each thought step) and must be echoed back unchanged when continuing on the same model - modified blocks are rejected, and dropping them can break a multi-turn tool loop. Prompt-elicited reasoning is inert text you can render or discard freely. That may mean the two need different payloads, not just a flag on ThinkingBlock:

sealed interface ThinkingBlock {
    data class Native(val content: String, val providerState: OpaqueReasoningState) : ThinkingBlock
    data class Prompted(val content: String, val tag: String) : ThinkingBlock
}

[igordayen]

@jorander @azanux - I did a good deal of regression testing of the combination recently:

streaming+tooling+reasoning.

Outcome: Spring AI totally swallows reasoning before the tool loop, and within the tool loop, only available reasoning is the final one. After the tool loop, see issues: #1800, #1799

Do you have in mind a reliable algorithm for how to split native vs. non-native?

Still thinking about a unified model with tagType=provider-name if possible. More coming....

[azanux]

@jorander @azanux - I did a good deal of regression testing of the combination recently:

streaming+tooling+reasoning.

Outcome: Spring AI totally swallows reasoning before the tool loop, and within the tool loop, only available reasoning is the final one. After the tool loop, see issues: #1800, #1799

Do you have in mind a reliable algorithm for how to split native vs. non-native?

Still thinking about a unified model with tagType=provider-name if possible. More coming....

not sure but , it seems to be solved in spring AI version 2.0 (at least for gemini)

[igordayen]

what about thinking(include=["think","reason"], exclude=["div"]) ?

[jorander]

@igordayen Wouldn't returning html or xml require a very long list of excluded tags not to be interpreted as thinking?

If so, I think either application level thinking extraction should be used only for when returning json-payloads and not for string content or the dynamic tag detection should be removed and replaced with .thinking(thinking tags vararg) so that it gets fully in the hands of the application code what tags to extract as thinking/reasoning. I lean towards the latter because it gets more predictable and all generation methods get the same functionality.