Skip to content

fix(api): render Anthropic prompts per architecture - #861

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
terrizoaguimor:fix/anthropic-architecture-rendering
Aug 7, 2026
Merged

fix(api): render Anthropic prompts per architecture#861
JustVugg merged 1 commit into
JustVugg:devfrom
terrizoaguimor:fix/anthropic-architecture-rendering

Conversation

@terrizoaguimor

Copy link
Copy Markdown
Contributor

Summary

  • route /v1/messages through the same architecture-aware chat renderer used by /v1/chat/completions
  • decode Inkling reasoning with its native content markers in both JSON and SSE responses
  • reject Anthropic tool requests on engines that do not implement tool use, instead of feeding them GLM prompt syntax
  • document the endpoint capability matrix for GLM, Inkling, Kimi K3, and DeepSeek V4

Problem

The OpenAI chat endpoint selected render_chat_inkling, render_chat_kimi, or render_chat_v4 from ARCH, but the Anthropic endpoint always called the GLM renderer. A request sent to Kimi K3, for example, reached the engine as:

[gMASK]<sop><|user|>Hi<|assistant|><think></think>

instead of its required length-framed payload:

K3CHAT1
M user 2
HiG 0

Kimi only invokes its native XTML builder when the payload begins with K3CHAT1\n; otherwise it tokenizes the GLM markers as ordinary text. Inkling and DeepSeek V4 similarly received a prompt from the wrong model family. Because GLM rendering accepted tools, Anthropic tool declarations also bypassed the explicit unsupported-parameter checks in the sibling renderers.

Anthropic response handling had the symmetric Inkling problem: it looked only for GLM </think> boundaries, not Inkling <|content_thinking|> / <|content_text|> markers.

Implementation

  • add one render_chat_for_arch dispatch point and use it from both protocol handlers
  • preserve Inkling audio plumbing through the shared renderer
  • use split_inkling for non-streaming Anthropic responses
  • use InklingStreamSplit for streaming responses and close the Anthropic thinking block before opening the text block
  • retain the existing GLM behavior and the Kimi/V4 response path

Tests

Added regressions that cover:

  • native prompt selection for all four served architectures
  • explicit tool rejection before generation for Inkling, Kimi K3, and DeepSeek V4
  • non-streaming Inkling reasoning/content separation
  • streaming Inkling marker splits across engine DATA boundaries
  • Anthropic event ordering: thinking block closes before the text block starts

Validation run locally on macOS:

python3 -m unittest tests.test_anthropic_messages tests.test_openai_server
Ran 143 tests in 64.962s
OK

make check
Ran 347 tests in 131.680s
OK (skipped=56)

The portable build was single-threaded because local libomp is unavailable; the build and full check suite still passed.

@kreuzzelg

Copy link
Copy Markdown
Contributor

Heads-up from #712, in the same spirit as JustVugg's note to us when #165 landed: our branch adds a render_chat_qwen and extends exactly the inline dispatch in chat_completion that this PR moves into render_chat_for_arch(). The conflict will be additive in intent — nothing contested — but it lands in the same lines, and render_chat_for_arch() will need a qwen entry that this PR (correctly, written against current dev) doesn't know about.

Whoever merges second: the resolution is one line — add

    renderer = (render_chat_kimi if ARCH == "kimi" else
                render_chat_qwen if ARCH == "qwen" else
                render_chat_v4 if ARCH == "deepseek_v4" else render_chat)

to the helper and keep both functions whole. If this PR goes first, we fold our entry into render_chat_for_arch() on our side, gladly — routing /v1/messages through the arch-aware renderer is exactly what the qwen engine wants too.

@JustVugg
JustVugg merged commit c6d860e into JustVugg:dev Aug 7, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants