Claude Agent SDK: _handle_result logs main-agent usage, under-counts turns with subagents

[lsuarez-ekline]

Summary

The Claude Agent SDK integration logs a turn's token metrics from ResultMessage.usage, which is the main-agent-only cumulative. On any turn that spawns subagents (the Task/Agent tool), the subagent tokens are billed but are missing from the trace, so the reported token total is a large under-count.

Where

braintrust/integrations/claude_agent_sdk/tracing.py, ContextTracker._handle_result — the only place token metrics are attached:

if hasattr(message, "usage"):
    usage_metrics, usage_metadata = extract_anthropic_usage(message.usage)
    ctx = self._get_context(None)
    if ctx.llm_span and (usage_metrics or usage_metadata):
        ctx.llm_span.log(metrics=usage_metrics or None, ...)

ResultMessage.usage is the cumulative usage for the main agent. ResultMessage.model_usage is the documented per-model breakdown that is per-agent — it includes subagent calls. The wrapper never reads model_usage.

Reproduction

Requires an Anthropic API key. The script prints the token total the wrapper would log (from usage) next to the authoritative model_usage total for the same turn.

# pip install claude-agent-sdk braintrust
# export ANTHROPIC_API_KEY=...
import anyio
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage
from braintrust.wrappers.claude_agent_sdk import setup_claude_agent_sdk

# Instrument the SDK exactly as a Braintrust user would.
setup_claude_agent_sdk()

PROMPT = (
    "Use the Task tool to launch 3 subagents in parallel. Each subagent should "
    "independently run several tool calls exploring a different topic, then report "
    "back. Wait for all three, then synthesize. Actually spawn them via the Task tool."
)

def usage_total(u: dict) -> int:
    return (u.get("input_tokens", 0) + u.get("output_tokens", 0)
            + u.get("cache_read_input_tokens", 0) + u.get("cache_creation_input_tokens", 0))

def model_usage_total(mu: dict) -> int:
    return sum(m.get("inputTokens", 0) + m.get("outputTokens", 0)
               + m.get("cacheReadInputTokens", 0) + m.get("cacheCreationInputTokens", 0)
               for m in mu.values())

async def main():
    result = None
    async for msg in query(
        prompt=PROMPT,
        options=ClaudeAgentOptions(permission_mode="bypassPermissions"),
    ):
        if isinstance(msg, ResultMessage):
            result = msg

    logged = usage_total(result.usage)              # what _handle_result logs today
    authoritative = model_usage_total(result.model_usage)  # per-agent, incl. subagents
    print(f"wrapper-logged (usage):  {logged:,}")
    print(f"model_usage (per-agent): {authoritative:,}")
    print(f"wrapper captures:        {logged / authoritative:.1%}")

anyio.run(main)

Observed (one run, 3 subagents): usage = 126,113 vs model_usage = 873,617 → the wrapper captures ~14.4% (two models: main Opus + Haiku subagents). In the logged trace the tokens land on a single LLM span; the subagent spans carry none. A run without subagents gives usage == model_usage (unaffected). Exact numbers vary per run; the direction does not.

Cross-check that model_usage is the correct one

model_usage × Anthropic list price reconstructs ResultMessage.total_cost_usd (the SDK's own reported cost) exactly; the usage-based total reconstructs only a fraction. So the under-count is in what's logged, not in model_usage.

Suggested fix

In _handle_result, derive the turn's token metrics from model_usage (aggregated across models) rather than usage — or attribute tokens onto the per-subagent spans. Anthropic's SDK documents model_usage as the authoritative per-agent source (anthropics/claude-agent-sdk-python#987).

PR: #565 — prefers the aggregated model_usage for the logged metrics, falls back to usage (metadata unchanged). Open to the per-subagent-span alternative if that's preferred.

Related

• Same class of bug in the .NET SDK: [BOT ISSUE] Anthropic streaming responses missing token usage metrics braintrust-sdk-dotnet#28, Port JS parameters implementation to Python #45
• model_usage as authoritative per-agent source: [Feature Request] Streaming per-subagent token usage events during execution anthropics/claude-agent-sdk-python#987