RATCHAW · RATCHAW · Jul 12, 2026 · Jul 12, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -4,7 +4,7 @@ extractkit is an open-source TypeScript document-extraction engine: Zod schema +
 
 ## Current Phase
 
-**Phase 3 (playground) shipped against mock models; the first live provider run is the remaining gate across evals + playground.** `packages/core`, the `packages/evals` harness, and `apps/playground` are all implemented and tested against mock models. The 25 CORD-v2 receipts are pinned in `packages/evals/data/manifest.json`; the 25 DocILE invoices are blocked on the dataset token (ROADMAP Phase 0, pending human action) — loader and curation script are ready. The eval lineup spans Anthropic, OpenAI, and Google Gemini (`packages/evals/src/models.ts`); a run includes every provider whose API key is set, or the subset named in `EVAL_PROVIDERS`. The playground (`apps/playground`) is a Hono API + Vite/React client: `GET /api/config`, `POST /api/extract` streamed as SSE via core's `streamExtract`, preset invoice/receipt schemas defined server-side, and a model registry resolved from whichever provider keys are present. It needs the same provider keys to run live, and its model registry is injectable so the routes test against a mock model. The first live run (any one of `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_GENERATIVE_AI_API_KEY`) doubles as core's first live-provider validation, fills the benchmark page + README table via `pnpm report`, and enables capturing the playground demo GIF. Keep README/ROADMAP/docs in sync with what actually ships.
+**The first live provider run has landed (OpenAI, receipts); the remaining gates are the other provider lineups, the DocILE invoice half, and the demo GIF.** `packages/core`, the `packages/evals` harness, and `apps/playground` are all implemented and tested. On 2026-07-12 the OpenAI lineup ran against the 25 pinned CORD-v2 receipts (`packages/evals/results/run-2026-07-12T12-17-58-095Z.json`), and `pnpm report` published those numbers to `docs/benchmark.md` + the README table — validating core's live provider path (the playground was separately confirmed live the same day). The 25 DocILE invoices remain blocked on the dataset token (ROADMAP Phase 0, pending human action) — loader and curation script are ready. The eval lineup spans Anthropic, OpenAI, and Google Gemini (`packages/evals/src/models.ts`); a run includes every provider whose API key is set (non-empty), or the subset named in `EVAL_PROVIDERS`. The playground (`apps/playground`) is a Hono API + Vite/React client: `GET /api/config`, `POST /api/extract` streamed as SSE via core's `streamExtract`, preset invoice/receipt schemas defined server-side, and a model registry resolved from whichever provider keys are present. It needs the same provider keys to run live, and its model registry is injectable so the routes test against a mock model. Still open before release: the Anthropic and Gemini lineups (set the keys and re-run `pnpm run-eval`), the DocILE invoice half, and capturing the playground demo GIF. Keep README/ROADMAP/docs in sync with what actually ships.
 
 ## Planned Architecture
 

diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 **Extraction you can audit.** Define a Zod schema, feed it a PDF or image, get back schema-validated JSON where every field carries provenance — the page and bounding box it came from — plus a confidence score.
 
-> **Status: in development.** The core library ([`packages/core`](./packages/core)), the eval harness ([`packages/evals`](./packages/evals)), and the playground ([`apps/playground`](./apps/playground)) are implemented and tested against mock models. What's left is the first live provider run, which publishes the benchmark table and the demo GIF. See [ROADMAP.md](./ROADMAP.md).
+> **Status: in development.** The core library ([`packages/core`](./packages/core)), the eval harness ([`packages/evals`](./packages/evals)), and the playground ([`apps/playground`](./apps/playground)) are implemented and tested. The first live provider run is in — the OpenAI lineup across the CORD-v2 receipt set — validating core's live path and publishing the [benchmark](#benchmark) below. Still open: the Anthropic and Gemini lineups, the DocILE invoice half (blocked on a dataset token), and the demo GIF. See [ROADMAP.md](./ROADMAP.md).
 
 ## Why
 
@@ -20,7 +20,13 @@ See [ROADMAP.md](./ROADMAP.md) for the build plan.
 
 <!-- benchmark:start -->
 
-*No results published yet — the first live eval run is pending. Numbers will appear here only from recorded runs; see [`packages/evals`](./packages/evals) to reproduce.*
+**Receipts — CORD-v2 (photographed shop receipts)**
+
+| Model | Docs | Field accuracy | Grounding hit@0.5 | Mean IoU | Cost / 1k docs |
+|---|---|---|---|---|---|
+| gpt-5.6-sol | 25 | 94.1% | 82.5% | 66.5% | $57.99 |
+| gpt-5.6-luna | 25 | 88.0% | 51.8% | 46.8% | $11.91 |
+| gpt-5.4-mini | 25 | 84.7% | 0.4% | 1.4% | $4.77 |
 
 <!-- benchmark:end -->
 

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -12,15 +12,15 @@
 - [x] OCR-failure handling, retries, streaming, cost tracking
 - [x] Vitest suite covering the failure paths, not just happy path (53 tests, mock models)
 
-Tested against mock models only so far; first live-provider validation happens when Phase 2 eval runs stand up.
+Live-provider path validated via the playground and the first Phase 2 eval run (OpenAI, 2026-07-12).
 
 ## Phase 2 — Evals (`packages/evals`)
 
 - [x] Harness: accuracy per field per model, grounding accuracy (predicted vs. ground-truth bbox), cost per 1k docs — tested against mock models; report generation from recorded runs
 - [x] Receipt half pinned: 25 CORD-v2 test docs curated by mapping-consistency checks, pinned by id + SHA-256 in `packages/evals/data/manifest.json`
 - [ ] Invoice half pinned: blocked on the DocILE token (Phase 0 human action); curation script is ready
-- [ ] First live eval run (needs a provider key — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and/or `GOOGLE_GENERATIVE_AI_API_KEY`) → benchmark page generated from real runs (engineering target: >90% field accuracy on the invoice set)
-- [ ] Benchmark table in README (markers in place; filled by `pnpm report` from a recorded run)
+- [x] First live eval run — OpenAI lineup on the CORD-v2 receipt set (2026-07-12); benchmark page + README table generated from the recorded run. Anthropic and Gemini lineups, and the DocILE invoice half (>90% field-accuracy engineering target), still to run.
+- [x] Benchmark table in README, filled by `pnpm report` from the recorded run
 
 ## Phase 3 — Playground (`apps/playground`)
 

diff --git a/docs/benchmark.md b/docs/benchmark.md
@@ -0,0 +1,53 @@
+# extractkit benchmark
+
+Field-level extraction accuracy, grounding accuracy, and cost, measured on pinned public
+documents: receipts from [CORD-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2)
+(NAVER CLOVA, CC BY 4.0) and invoices from [DocILE](https://docile.rossum.ai/) (Rossum).
+Documents are pinned by id + checksum in
+[`packages/evals/data/manifest.json`](../packages/evals/data/manifest.json) and fetched from
+their canonical hosts at eval time — see [`packages/evals`](../packages/evals) to reproduce.
+
+Run started 2026-07-12T12:17:58.095Z. Manifest checksum `3fcd95f11b94`.
+
+**Metrics.** *Field accuracy*: fraction of ground-truth fields whose predicted value matches
+under normalized comparison (whitespace/case for text, digits-and-sign for amounts, lenient
+numeric parse for quantities); a field the document does not carry counts as correct only when
+the model returned null for it. *Grounding hit@0.5*: among fields with a correct value and a
+ground-truth region, the fraction where the predicted bounding box overlaps an annotated region
+with IoU ≥ 0.5 on the right page; a missing bbox or wrong page scores 0. *Cost / 1k docs*:
+measured token usage priced at published per-token rates.
+
+## Receipts — CORD-v2 (photographed shop receipts)
+
+| Model | Docs | Field accuracy | Grounding hit@0.5 | Mean IoU | Cost / 1k docs |
+|---|---|---|---|---|---|
+| gpt-5.6-sol | 25 | 94.1% | 82.5% | 66.5% | $57.99 |
+| gpt-5.6-luna | 25 | 88.0% | 51.8% | 46.8% | $11.91 |
+| gpt-5.4-mini | 25 | 84.7% | 0.4% | 1.4% | $4.77 |
+
+### Accuracy per field
+
+| Field | gpt-5.6-sol | gpt-5.6-luna | gpt-5.4-mini |
+|---|---|---|---|
+| `discount` | 96.0% (25) | 100.0% (25) | 92.0% (25) |
+| `lineItems[].amount` | 91.6% (83) | 84.3% (83) | 92.8% (83) |
+| `lineItems[].description` | 88.0% (83) | 72.3% (83) | 78.3% (83) |
+| `lineItems[].quantity` | 97.6% (83) | 95.2% (83) | 97.6% (83) |
+| `lineItems[].unitPrice` | 98.8% (83) | 96.4% (83) | 62.7% (83) |
+| `serviceCharge` | 96.0% (25) | 96.0% (25) | 88.0% (25) |
+| `subtotal` | 92.0% (25) | 80.0% (25) | 84.0% (25) |
+| `tax` | 96.0% (25) | 92.0% (25) | 92.0% (25) |
+| `total` | 92.0% (25) | 84.0% (25) | 92.0% (25) |
+
+## Caveats
+
+- Both datasets are public and widely cited; frontier models have likely seen them in training.
+  Read these numbers as a comparative measurement across models under identical conditions, not
+  an absolute capability claim.
+- CORD receipts are Indonesian (Latin script); the numbers are not universal across locales.
+- Line items are aligned to ground truth by printed order; a correct item at the wrong position
+  scores as wrong.
+
+Receipt data © NAVER CLOVA, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), via the
+official [cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) dataset. DocILE
+documents are not redistributed; runners fetch them with their own access token.