feat(evals): add Promptfoo-based AI tool call evaluation suite

[tupizz]

Summary

Adds an automated evaluation suite for validating LLM tool call quality across SuperDoc's Document Engine API. This is the foundation for measuring and improving how well models use our tools.

What it does

Given a document editing task (e.g., "Find the indemnification clause and rewrite it"), the suite checks:

1. Did the model call the right tool? (query_match, not get_document_text)
2. Did it pass correct arguments? (select.type: "text", not a bare string)
3. Did it follow production rules? (no mixed rewrite + format batches, correct require values)

Architecture

User prompt          "Replace all mentions of Company A with Acme Corp"
      |
      v
Promptfoo sends      prompt + 6 tool definitions --> LLM (GPT-4o, GPT-5.4, etc.)
to the LLM                    |
                              v
LLM returns          [{ function: { name: "query_match", arguments: {...} } }]
tool calls                    |
                              v
Assertions check     tool-call-f1 + file://lib/assertions.cjs:validOpNames
the output                    |
                              v
Result               PASS / FAIL with reason

Three eval configs for different purposes

promptfooconfig.yaml                 Main suite: 4 OpenAI models, 25 deterministic tests
                                     Cost: ~$0.30 per run. Cached re-runs: free.

promptfooconfig.cross-provider.yaml  Cross-provider: GPT-5.4 vs Claude vs Gemini
                                     Tests with BOTH full prompt and minimal prompt
                                     to measure system prompt value.

promptfooconfig.gdpval.yaml          GDPval benchmark: Model+SuperDoc vs Model-Only
                                     Uses llm-rubric (costs ~$1-2 per run).

File structure

evals/
  promptfooconfig.yaml                 Configs (root level, Promptfoo convention)
  promptfooconfig.cross-provider.yaml
  promptfooconfig.gdpval.yaml

  prompts/                             What we send to LLMs
    agent.txt                          Full system prompt (from labs agent)
    minimal.txt                        Minimal prompt (customer simulation)

  tests/                               What we check
    tool-tests.yaml                    15 tests: tool selection + args + correctness
    workflows.yaml                     11 tests: find/replace, tracked changes, lists
    cross-provider.yaml                10 tests: realistic customer prompts
    gdpval-workflows.yaml              5 tests: Model+SuperDoc vs baseline

  lib/                                 Helper code
    assertions.cjs                     Shared assertion functions (15 exports)
    normalize.cjs                      Cross-provider output normalization
    extract.mjs                        Tool extraction from SDK artifacts
    save-baseline.mjs                  Save versioned result snapshot
    compare-baselines.mjs              Compare two snapshots

How tools are loaded

Tool definitions come from the SDK-generated packages/sdk/tools/tools.openai.json. The extraction script reads tools-policy.json for the essential tool list and writes a subset to lib/essential.json (gitignored). This keeps tool definitions DRY and automatically picks up upstream changes.

packages/sdk/tools/tools-policy.json  -->  lib/extract.mjs  -->  lib/essential.json
packages/sdk/tools/tools.openai.json       (reads both)          (6 tools, gitignored)

Cross-provider output normalization

LLM providers return tool calls in different formats:

• OpenAI: [{function: {name, arguments}}]
• Anthropic: {type: "tool_use", name, input}
• Google: [{functionCall: {name, args}}]

lib/normalize.cjs converts all formats to OpenAI's array format so assertions work across providers.

Test results (initial baseline)

Model	Pass Rate	Notes
GPT-4o	100% (25/25)	Best tool-calling accuracy
GPT-4.1	100% (25/25)	Matches GPT-4o with tool_choice: required
GPT-5.4	96% (24/25)	Calls get_document_text for list ops
GPT-4.1-mini	96% (24/25)	Uses text.insert for headings

Cross-provider results (minimal vs full prompt)

Model + Prompt	Pass Rate
GPT-5.4 + Full prompt	100%
GPT-5.4 + Minimal prompt	80%
Claude Sonnet 4.6 + Full prompt	80%
Gemini 2.5 Pro + Minimal prompt	70%
Claude Sonnet 4.6 + Minimal prompt	40%

Key finding: Our system prompt doubles Claude's accuracy (40% to 80%). This proves the value of our tool documentation.

Why Promptfoo

Evaluated Promptfoo, Braintrust, DeepEval, Langfuse, and OpenAI Evals. Promptfoo won because:

• TypeScript native (matches our stack)
• YAML-driven (no code to run tests)
• Built-in tool-call-f1 assertion type
• Caching (re-runs after assertion changes are free)
• Web UI for inspecting results
• GitHub Actions support for CI

Commands

pnpm run extract-tools   # Extract tools from SDK (run once)
pnpm run eval            # Run main suite (~$0.30)
pnpm run eval:cross      # Cross-provider comparison
pnpm run eval:gdpval     # GDPval benchmark (~$1-2)
pnpm run eval:view       # Open web UI

Test plan

• pnpm run extract-tools extracts 6 tools
• pnpm run eval passes 97%+ on 4 OpenAI models
• pnpm run eval:cross runs across 3 providers
• Assertions produce descriptive failure reasons
• Cached re-runs are instant and free
• CI workflow (future PR)