[Bug]: Eagle3 speculative decoding silently drops tool_calls for gpt-oss on trtllm-serve

[NPB2026]

System Info

[Bug] Eagle3 speculative decoding silently drops tool_calls for gpt-oss on trtllm-serve

Summary

With gpt-oss-120b on trtllm-serve, enabling Eagle3 speculative decoding makes the server
return an empty tool_calls array and empty content, even though the model clearly decided
to call the function (its reasoning_content says so, and it emitted the tokens).

Disabling speculative decoding — changing nothing else — makes tool calling work correctly.

The Harmony adapter itself is active in both cases (reasoning_content is populated either way), so
this is not a model-detection or chat-template problem: the tool-call channel is lost specifically
when Eagle3 is on.

This may be the missing link between #8615 and #10612

Two open issues describe what look like the same underlying failure, seen from different angles:

• [Bug]: Infinite loop using speculative decoding gpt-oss-120b-Eagle3 on 1.2.0rc1 container and RTX Pro 6000 with Jan AI as UI #8615 (Investigating) — speculative decoding with gpt-oss-120b-Eagle3 produces corrupted
  channel tokens: <|channel|>!!!!!!!!!!!!... and an infinite loop.
• [Bug]: Structured output constantly misses proper channel establishment for batch and streaming and result is often malformed. #10612 (open) — the Harmony channel-establishment sequence
  (<|channel|>final<|message|>, token IDs 200005, 17196, 200008) is not generated, the
  parser hits "Unexpected EOS while waiting for message header to complete", and falls back to
  corrupted/empty output.

Our A/B adds the missing variable: Eagle3 is the trigger, and the damage lands on the Harmony
control tokens — which are exactly the rare, structural tokens a draft head is worst at predicting
(<|channel|>, <|message|>).

Our failure mode is the most dangerous of the three, because it is completely silent. #8615 loops
loudly; #10612 returns visibly malformed JSON. Here: HTTP 200, finish_reason: "stop", no warning —
the tool call simply is not there. A framework cannot tell this apart from "the model chose not to call
the tool".

(#7163 is closed and used xgrammar with no speculative_config, so it is a different path.)

Environment

Container	nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13
GPU	NVIDIA GB10 (DGX Spark), compute capability sm_121
Driver	580.159.03
Backend	--backend pytorch, --tp_size 1
Model	openai/gpt-oss-120b (MXFP4)
Draft model	nvidia/gpt-oss-120b-Eagle3-long-context

Reproduction

A — WITH Eagle3 → tool_calls is empty (bug)

extra-llm-api-config.yml:

enable_attention_dp: false
disable_overlap_scheduler: false
enable_autotuner: false
cuda_graph_config:
    max_batch_size: 4
speculative_config:
    decoding_type: Eagle
    max_draft_len: 5
    speculative_model_dir: /opt/gpt-oss-120b-Eagle3/
kv_cache_config:
    free_gpu_memory_fraction: 0.9
    enable_block_reuse: false

trtllm-serve openai/gpt-oss-120b \
  --tokenizer <local snapshot path> \
  --backend pytorch --tp_size 1 --max_batch_size 4 \
  --host 0.0.0.0 --port 8005 \
  --extra_llm_api_options extra-llm-api-config.yml

Request:

curl -s http://127.0.0.1:8005/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"openai/gpt-oss-120b",
  "messages":[{"role":"user","content":"What is the weather in Rome? Use the tool."}],
  "tools":[{"type":"function","function":{
      "name":"get_weather","description":"Get the weather of a city",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
  "tool_choice":"auto","max_tokens":300}'

Response (bug):

{
  "message": {
    "role": "assistant",
    "content": "",
    "reasoning_content": "User asks weather in Rome. Use function.",
    "tool_calls": []
  },
  "finish_reason": "stop",
  "usage": { "completion_tokens": 37 }
}

Note: completion_tokens: 37 while the reasoning text is only ~10 words — the tool-call tokens were
generated and then dropped. reasoning_content being populated proves the Harmony adapter is active
(openai_server.py: use_harmony = (model_config.model_type == "gpt_oss")).

B — WITHOUT Eagle3 → works

Same command, same image, same model, same tokenizer, same request. Only the config differs — the
speculative_config block is removed:

enable_attention_dp: false
disable_overlap_scheduler: false
enable_autotuner: false
cuda_graph_config:
    max_batch_size: 4
kv_cache_config:
    free_gpu_memory_fraction: 0.9
    enable_block_reuse: false

Response (correct):

{
  "tool_calls": [
    { "function": { "name": "get_weather", "arguments": "{\"city\": \"Rome\"}" } }
  ]
}

Why this matters

The failure is silent: HTTP 200, finish_reason: "stop", no warning in the logs. An agent
framework sees an assistant message with empty content and no tool call, and simply believes the model
declined to act. Any multi-agent system built on tool calling will appear to "work" while doing
nothing — which is how we spent hours blaming the model.

Also worth noting: the DGX Spark playbook
(dgx-spark-playbooks/nvidia/speculative-decoding, Option 1) recommends exactly this Eagle3 recipe for
gpt-oss-120b on a single Spark. Anyone following it and then using tools will hit this.

Suggested area to look at

tensorrt_llm/serve/harmony_adapter.py — the token-batch path
(process_token_batch / _get_or_create_tool_call) versus how tokens are delivered when
Eagle3OneModelSampler is in use. The reasoning channel survives; the tool-call channel does not.

A loud error — or even a startup warning that tool calling is unsupported with speculative decoding —
would already be a large improvement over silently returning an empty tool_calls.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Container	nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13
GPU	NVIDIA GB10 (DGX Spark), compute capability sm_121
Driver	580.159.03
Backend	--backend pytorch, --tp_size 1
Model	openai/gpt-oss-120b (MXFP4)
Draft model	nvidia/gpt-oss-120b-Eagle3-long-context

A — WITH Eagle3 → tool_calls is empty (bug)

config.yml:

enable_attention_dp: false
disable_overlap_scheduler: false
enable_autotuner: false
cuda_graph_config:
    max_batch_size: 4
speculative_config:
    decoding_type: Eagle
    max_draft_len: 5
    speculative_model_dir: /opt/gpt-oss-120b-Eagle3/
kv_cache_config:
    free_gpu_memory_fraction: 0.9
    enable_block_reuse: false

Server:

trtllm-serve openai/gpt-oss-120b \
  --tokenizer <path to the local HF snapshot> \
  --backend pytorch --tp_size 1 --max_batch_size 4 \
  --host 0.0.0.0 --port 8000 \
  --extra_llm_api_options config.yml

Request:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "openai/gpt-oss-120b",
  "messages": [{"role": "user", "content": "What is the weather in Rome? Use the tool."}],
  "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get the weather of a city",
      "parameters": {"type": "object",
                     "properties": {"city": {"type": "string"}},
                     "required": ["city"]}}}],
  "tool_choice": "auto",
  "max_tokens": 300}'

B — WITHOUT Eagle3 → works

Same image, same model, same tokenizer, same request. Only the speculative_config block is removed
from config.yml.

---


### Expected behavior

```markdown
The response should contain the tool call the model decided to make — which is exactly what happens
when Eagle3 is disabled:

```json
{
  "message": {
    "role": "assistant",
    "tool_calls": [
      {"function": {"name": "get_weather", "arguments": "{\"city\": \"Rome\"}"}}
    ]
  }
}

---


### actual behavior

```markdown
With Eagle3 enabled, the tool call is silently lost:

```json
{
  "message": {
    "role": "assistant",
    "content": "",
    "reasoning_content": "User asks weather in Rome. Use function.",
    "tool_calls": []
  },
  "finish_reason": "stop",
  "usage": { "completion_tokens": 37 }
}

Three things are worth pointing out:

1. The model did decide to call the function — reasoning_content literally says "Use function."
2. completion_tokens: 37, while the reasoning text is only ~10 words. The tool-call tokens were
   generated and then dropped.
3. The Harmony adapter is active — reasoning_content being populated proves it
   (openai_server.py: use_harmony = (model_config.model_type == "gpt_oss")). So this is not a
   model-detection or chat-template problem.

There is no error, no warning, and finish_reason is "stop". The response is a perfectly valid
HTTP 200. An agent framework cannot distinguish this from "the model chose not to call the tool".

---


### additional notes

```markdown
### This may be the missing link between #8615 and #10612

Two **open** issues look like the same underlying failure seen from different angles:

- **#8615** (*Investigating*) — speculative decoding with `gpt-oss-120b-Eagle3` produces **corrupted
  channel tokens** (`<|channel|>!!!!!!!!!!!!…`) and an infinite loop.
- **#10612** (*open*) — the Harmony **channel-establishment sequence**
  (`<|channel|>final<|message|>`, token IDs `200005`, `17196`, `200008`) **is not generated**, the
  parser hits *"Unexpected EOS while waiting for message header to complete"*, and falls back to
  corrupted or empty output.

The A/B above adds the missing variable: **Eagle3 is the trigger**, and the damage lands on the Harmony
**control tokens** — precisely the rare, structural tokens (`<|channel|>`, `<|message|>`) that a draft
head is worst at predicting.

**This failure mode is the most dangerous of the three, because it is completely silent.** #8615 loops
loudly; #10612 returns visibly malformed JSON. Here everything looks fine and the tool call is simply
absent.

(#7163 is closed and used xgrammar with **no** `speculative_config` — a different path.)

### Suggested area to look at

`tensorrt_llm/serve/harmony_adapter.py` — the token-batch path (`process_token_batch` /
`_get_or_create_tool_call`) versus how tokens are delivered when `Eagle3OneModelSampler` is in use.
The reasoning channel survives; the tool-call channel does not.

### Why this matters beyond one user

The official DGX Spark playbook (`dgx-spark-playbooks/nvidia/speculative-decoding`, Option 1)
recommends **exactly this Eagle3 recipe** for gpt-oss-120b on a single Spark. Anyone who follows it and
then uses tools will hit this — and, because it is silent, will likely blame the model.

Even just a startup warning ("tool calling is not supported with speculative decoding") would be a large
improvement over returning an empty `tool_calls` with HTTP 200.

---

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.