Skip to content

evals: Fail the run on an API auth error instead of scoring it as an agent miss #1526

Open

Description

@jirispilka

Problem

With an invalid ANTHROPIC_API_KEY, the Claude Code CLI returns a result whose text is Invalid API key · Fix external API key after about 3 s and no model turn. The harness treats it as a normal conversation: kind: "tool-call" items score first_tool_match 0 — no tool call attempted, kind: "agent" items go to the judge, which fails them for "not attempting any tool calls". The run reports 0/125 trials passed (pass_rate 0.00) and a real-looking Langfuse dataset run.

Seen on 2026-10-12: CI (master evals) runs 41–43 (11e7e7c, 22aa888, 20fef3b) and the PR-tier job on #1516 (e08b4e0), both tiers. The judge, on the CLI login (--claude-judge), was unaffected.

Expected

An auth failure is a harness failure under evals/README.md decision 4: throw from the agent run (not transient, so no retry), 🔥 Never completed, no judge call, no scores, Langfuse trace ERROR with the message. Same shape as the "MCP server not connected at init" guard in #1519.

Notes

  • Detect it where the CLI reports it (the result message's subtype/text, or an assistant turn whose only content is the auth error); confirm the exact shape against the pinned CLI before pinning a string.
  • A unit test that feeds the CLI's auth-error result through runAgentConversation and asserts the throw, plus the isTransientAgentError false case.

Found while watching CI on #1516.

Activity

  1. QamerZaman123 commented on Oct 13, 2026

    @QamerZaman123

    Hi, @jirispilka, can I work on this issue? This will be my first contribution to Apify

  2. jirispilka commented on Oct 13, 2026

    @jirispilka
    CollaboratorAuthor
    1. Hi, @jirispilka, can I work on this issue? This will be my first contribution to Apify

    Hi, thanks, bu it is running internal evals, we will handle it, but thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions