You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
evals: Fail the run on an API auth error instead of scoring it as an agent miss #1526
With an invalid ANTHROPIC_API_KEY, the Claude Code CLI returns a result whose text is Invalid API key · Fix external API key after about 3 s and no model turn. The harness treats it as a normal conversation: kind: "tool-call" items score first_tool_match 0 — no tool call attempted, kind: "agent" items go to the judge, which fails them for "not attempting any tool calls". The run reports 0/125 trials passed (pass_rate 0.00) and a real-looking Langfuse dataset run.
Seen on 2026-10-12: CI (master evals) runs 41–43 (11e7e7c, 22aa888, 20fef3b) and the PR-tier job on #1516 (e08b4e0), both tiers. The judge, on the CLI login (--claude-judge), was unaffected.
Expected
An auth failure is a harness failure under evals/README.md decision 4: throw from the agent run (not transient, so no retry), 🔥 Never completed, no judge call, no scores, Langfuse trace ERROR with the message. Same shape as the "MCP server not connected at init" guard in #1519.
Notes
Detect it where the CLI reports it (the result message's subtype/text, or an assistant turn whose only content is the auth error); confirm the exact shape against the pinned CLI before pinning a string.
A unit test that feeds the CLI's auth-error result through runAgentConversation and asserts the throw, plus the isTransientAgentError false case.
Activity
QamerZaman123 commentedon Oct 13, 2026
Hi, @jirispilka, can I work on this issue? This will be my first contribution to Apify
jirispilka commentedon Oct 13, 2026
Hi, thanks, bu it is running internal evals, we will handle it, but thanks!