diff --git a/CLAUDE.md b/CLAUDE.md index 270b8e4..c128a76 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ extractkit is an open-source TypeScript document-extraction engine: Zod schema + ## Current Phase -**Phase 3 (playground) shipped against mock models; the first live provider run is the remaining gate across evals + playground.** `packages/core`, the `packages/evals` harness, and `apps/playground` are all implemented and tested against mock models. The 25 CORD-v2 receipts are pinned in `packages/evals/data/manifest.json`; the 25 DocILE invoices are blocked on the dataset token (ROADMAP Phase 0, pending human action) — loader and curation script are ready. The eval lineup spans Anthropic, OpenAI, and Google Gemini (`packages/evals/src/models.ts`); a run includes every provider whose API key is set, or the subset named in `EVAL_PROVIDERS`. The playground (`apps/playground`) is a Hono API + Vite/React client: `GET /api/config`, `POST /api/extract` streamed as SSE via core's `streamExtract`, preset invoice/receipt schemas defined server-side, and a model registry resolved from whichever provider keys are present. It needs the same provider keys to run live, and its model registry is injectable so the routes test against a mock model. The first live run (any one of `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_GENERATIVE_AI_API_KEY`) doubles as core's first live-provider validation, fills the benchmark page + README table via `pnpm report`, and enables capturing the playground demo GIF. Keep README/ROADMAP/docs in sync with what actually ships. +**The first live provider run has landed (OpenAI, receipts); the remaining gates are the other provider lineups, the DocILE invoice half, and the demo GIF.** `packages/core`, the `packages/evals` harness, and `apps/playground` are all implemented and tested. On 2026-07-12 the OpenAI lineup ran against the 25 pinned CORD-v2 receipts (`packages/evals/results/run-2026-07-12T12-17-58-095Z.json`), and `pnpm report` published those numbers to `docs/benchmark.md` + the README table — validating core's live provider path (the playground was separately confirmed live the same day). The 25 DocILE invoices remain blocked on the dataset token (ROADMAP Phase 0, pending human action) — loader and curation script are ready. The eval lineup spans Anthropic, OpenAI, and Google Gemini (`packages/evals/src/models.ts`); a run includes every provider whose API key is set (non-empty), or the subset named in `EVAL_PROVIDERS`. The playground (`apps/playground`) is a Hono API + Vite/React client: `GET /api/config`, `POST /api/extract` streamed as SSE via core's `streamExtract`, preset invoice/receipt schemas defined server-side, and a model registry resolved from whichever provider keys are present. It needs the same provider keys to run live, and its model registry is injectable so the routes test against a mock model. Still open before release: the Anthropic and Gemini lineups (set the keys and re-run `pnpm run-eval`), the DocILE invoice half, and capturing the playground demo GIF. Keep README/ROADMAP/docs in sync with what actually ships. ## Planned Architecture diff --git a/README.md b/README.md index d922760..2062fbf 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ **Extraction you can audit.** Define a Zod schema, feed it a PDF or image, get back schema-validated JSON where every field carries provenance — the page and bounding box it came from — plus a confidence score. -> **Status: in development.** The core library ([`packages/core`](./packages/core)), the eval harness ([`packages/evals`](./packages/evals)), and the playground ([`apps/playground`](./apps/playground)) are implemented and tested against mock models. What's left is the first live provider run, which publishes the benchmark table and the demo GIF. See [ROADMAP.md](./ROADMAP.md). +> **Status: in development.** The core library ([`packages/core`](./packages/core)), the eval harness ([`packages/evals`](./packages/evals)), and the playground ([`apps/playground`](./apps/playground)) are implemented and tested. The first live provider run is in — the OpenAI lineup across the CORD-v2 receipt set — validating core's live path and publishing the [benchmark](#benchmark) below. Still open: the Anthropic and Gemini lineups, the DocILE invoice half (blocked on a dataset token), and the demo GIF. See [ROADMAP.md](./ROADMAP.md). ## Why @@ -20,7 +20,13 @@ See [ROADMAP.md](./ROADMAP.md) for the build plan. -*No results published yet — the first live eval run is pending. Numbers will appear here only from recorded runs; see [`packages/evals`](./packages/evals) to reproduce.* +**Receipts — CORD-v2 (photographed shop receipts)** + +| Model | Docs | Field accuracy | Grounding hit@0.5 | Mean IoU | Cost / 1k docs | +|---|---|---|---|---|---| +| gpt-5.6-sol | 25 | 94.1% | 82.5% | 66.5% | $57.99 | +| gpt-5.6-luna | 25 | 88.0% | 51.8% | 46.8% | $11.91 | +| gpt-5.4-mini | 25 | 84.7% | 0.4% | 1.4% | $4.77 | diff --git a/ROADMAP.md b/ROADMAP.md index f9df8c7..eebaf27 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -12,15 +12,15 @@ - [x] OCR-failure handling, retries, streaming, cost tracking - [x] Vitest suite covering the failure paths, not just happy path (53 tests, mock models) -Tested against mock models only so far; first live-provider validation happens when Phase 2 eval runs stand up. +Live-provider path validated via the playground and the first Phase 2 eval run (OpenAI, 2026-07-12). ## Phase 2 — Evals (`packages/evals`) - [x] Harness: accuracy per field per model, grounding accuracy (predicted vs. ground-truth bbox), cost per 1k docs — tested against mock models; report generation from recorded runs - [x] Receipt half pinned: 25 CORD-v2 test docs curated by mapping-consistency checks, pinned by id + SHA-256 in `packages/evals/data/manifest.json` - [ ] Invoice half pinned: blocked on the DocILE token (Phase 0 human action); curation script is ready -- [ ] First live eval run (needs a provider key — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and/or `GOOGLE_GENERATIVE_AI_API_KEY`) → benchmark page generated from real runs (engineering target: >90% field accuracy on the invoice set) -- [ ] Benchmark table in README (markers in place; filled by `pnpm report` from a recorded run) +- [x] First live eval run — OpenAI lineup on the CORD-v2 receipt set (2026-07-12); benchmark page + README table generated from the recorded run. Anthropic and Gemini lineups, and the DocILE invoice half (>90% field-accuracy engineering target), still to run. +- [x] Benchmark table in README, filled by `pnpm report` from the recorded run ## Phase 3 — Playground (`apps/playground`) diff --git a/docs/benchmark.md b/docs/benchmark.md new file mode 100644 index 0000000..8bf7b9d --- /dev/null +++ b/docs/benchmark.md @@ -0,0 +1,53 @@ +# extractkit benchmark + +Field-level extraction accuracy, grounding accuracy, and cost, measured on pinned public +documents: receipts from [CORD-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) +(NAVER CLOVA, CC BY 4.0) and invoices from [DocILE](https://docile.rossum.ai/) (Rossum). +Documents are pinned by id + checksum in +[`packages/evals/data/manifest.json`](../packages/evals/data/manifest.json) and fetched from +their canonical hosts at eval time — see [`packages/evals`](../packages/evals) to reproduce. + +Run started 2026-07-12T12:17:58.095Z. Manifest checksum `3fcd95f11b94`. + +**Metrics.** *Field accuracy*: fraction of ground-truth fields whose predicted value matches +under normalized comparison (whitespace/case for text, digits-and-sign for amounts, lenient +numeric parse for quantities); a field the document does not carry counts as correct only when +the model returned null for it. *Grounding hit@0.5*: among fields with a correct value and a +ground-truth region, the fraction where the predicted bounding box overlaps an annotated region +with IoU ≥ 0.5 on the right page; a missing bbox or wrong page scores 0. *Cost / 1k docs*: +measured token usage priced at published per-token rates. + +## Receipts — CORD-v2 (photographed shop receipts) + +| Model | Docs | Field accuracy | Grounding hit@0.5 | Mean IoU | Cost / 1k docs | +|---|---|---|---|---|---| +| gpt-5.6-sol | 25 | 94.1% | 82.5% | 66.5% | $57.99 | +| gpt-5.6-luna | 25 | 88.0% | 51.8% | 46.8% | $11.91 | +| gpt-5.4-mini | 25 | 84.7% | 0.4% | 1.4% | $4.77 | + +### Accuracy per field + +| Field | gpt-5.6-sol | gpt-5.6-luna | gpt-5.4-mini | +|---|---|---|---| +| `discount` | 96.0% (25) | 100.0% (25) | 92.0% (25) | +| `lineItems[].amount` | 91.6% (83) | 84.3% (83) | 92.8% (83) | +| `lineItems[].description` | 88.0% (83) | 72.3% (83) | 78.3% (83) | +| `lineItems[].quantity` | 97.6% (83) | 95.2% (83) | 97.6% (83) | +| `lineItems[].unitPrice` | 98.8% (83) | 96.4% (83) | 62.7% (83) | +| `serviceCharge` | 96.0% (25) | 96.0% (25) | 88.0% (25) | +| `subtotal` | 92.0% (25) | 80.0% (25) | 84.0% (25) | +| `tax` | 96.0% (25) | 92.0% (25) | 92.0% (25) | +| `total` | 92.0% (25) | 84.0% (25) | 92.0% (25) | + +## Caveats + +- Both datasets are public and widely cited; frontier models have likely seen them in training. + Read these numbers as a comparative measurement across models under identical conditions, not + an absolute capability claim. +- CORD receipts are Indonesian (Latin script); the numbers are not universal across locales. +- Line items are aligned to ground truth by printed order; a correct item at the wrong position + scores as wrong. + +Receipt data © NAVER CLOVA, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), via the +official [cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) dataset. DocILE +documents are not redistributed; runners fetch them with their own access token. diff --git a/packages/evals/results/run-2026-07-12T12-17-58-095Z.json b/packages/evals/results/run-2026-07-12T12-17-58-095Z.json new file mode 100644 index 0000000..8d368a8 --- /dev/null +++ b/packages/evals/results/run-2026-07-12T12-17-58-095Z.json @@ -0,0 +1,13485 @@ +{ + "startedAt": "2026-07-12T12:17:58.095Z", + "manifestChecksum": "3fcd95f11b94a49078731a6285f0024a8fa2a583ca49102b88efdd0a4c9d59d6", + "runs": [ + { + "model": "gpt-5.6-sol", + "docs": [ + { + "docId": "cord/test/0", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "-TICKET CP", + "predicted": "-TICKET CP", + "valueCorrect": true, + "iou": 0.7878445141806046, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.7330704530853087, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "60,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60.000", + "predicted": "60,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "60.000", + "predicted": "60.000", + "valueCorrect": true, + "iou": 0.8043284668022246, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "-60.000", + "predicted": "-60,000", + "valueCorrect": true, + "iou": 0.8481068674990949, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "5.455", + "predicted": "5.455", + "valueCorrect": true, + "iou": 0.8084320761467008, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "60.000", + "predicted": "60.000", + "valueCorrect": true, + "iou": 0.7459756257214283, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1758, + "outputTokens": 869, + "modelCalls": 1, + "costUSD": 0.03486 + } + }, + { + "docId": "cord/test/1", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "J.STB PROMO", + "predicted": "J.SIB PROMO", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17500", + "predicted": "17500", + "valueCorrect": true, + "iou": 0.8182569138970346, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Y.B.BAT", + "predicted": "Y.B.BAT", + "valueCorrect": true, + "iou": 0.774492537812078, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "46000", + "predicted": "46000", + "valueCorrect": true, + "iou": 0.8689757683166882, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Y.BASO PROM", + "predicted": "Y.BASO PROM", + "valueCorrect": true, + "iou": 0.7839984542519954, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "27500", + "predicted": "27500", + "valueCorrect": true, + "iou": 0.8653713527851444, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "91000", + "predicted": "91000", + "valueCorrect": true, + "iou": 0.8051637344736059, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 822, + "modelCalls": 1, + "costUSD": 0.03889 + } + }, + { + "docId": "cord/test/2", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "JASMINE MT ( L )", + "predicted": "JASMINE MT ( L )", + "valueCorrect": true, + "iou": 0.8693486968355724, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6653230495635459, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24,000", + "valueCorrect": true, + "iou": 0.694641830790148, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COCONUT JELLY ( L )", + "predicted": "COCONUT JELLY ( L )", + "valueCorrect": true, + "iou": 0.8832837593865817, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "4,000", + "predicted": "4,000", + "valueCorrect": true, + "iou": 0.6747367809702399, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,000", + "predicted": "28,000", + "valueCorrect": true, + "iou": 0.8318495172716172, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "28,000", + "predicted": "28,000", + "valueCorrect": true, + "iou": 0.732469204049469, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1144, + "modelCalls": 1, + "costUSD": 0.04799 + } + }, + { + "docId": "cord/test/3", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "DONAT GULA", + "predicted": "DONAT GULA", + "valueCorrect": true, + "iou": 0.47278266662961094, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1X", + "predicted": "1X", + "valueCorrect": true, + "iou": 0.5427125550006695, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@11000", + "predicted": "@11000", + "valueCorrect": true, + "iou": 0.5790159189477628, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11000", + "valueCorrect": true, + "iou": 0.5431629056698406, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "11,000", + "predicted": "11000", + "valueCorrect": true, + "iou": 0.5775756646818947, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "11,000", + "predicted": "11000", + "valueCorrect": true, + "iou": 0.4698019999766167, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 820, + "modelCalls": 1, + "costUSD": 0.03827 + } + }, + { + "docId": "cord/test/4", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "ICE BLACKCOFFE", + "predicted": "ICE BLACKCOFFE", + "valueCorrect": true, + "iou": 0.8137150746813178, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.8118960401069847, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "82,000", + "predicted": "82,000", + "valueCorrect": true, + "iou": 0.6695727201249178, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AVOCADO COFFEE", + "predicted": "AVOCADO COFFEE", + "valueCorrect": true, + "iou": 0.7717693284333492, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.810092704270734, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "61,000", + "predicted": "61,000", + "valueCorrect": true, + "iou": 0.6691683685339812, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CHIKEN KATSU FF", + "predicted": "CHICKEN KATSU FF", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.42068648254910107, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "51,000", + "predicted": "51,000", + "valueCorrect": true, + "iou": 0.6406116568507002, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "194,000", + "predicted": "194,000", + "valueCorrect": true, + "iou": 0.8267558742899279, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "19,400", + "predicted": "19,400", + "valueCorrect": true, + "iou": 0.6918119655210302, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "174,600", + "predicted": "174,600", + "valueCorrect": true, + "iou": 0.7788009847284316, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1299, + "modelCalls": 1, + "costUSD": 0.05264 + } + }, + { + "docId": "cord/test/5", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TRAD KY TOAST CARTE", + "predicted": "TRAD KY TOAST CARTE", + "valueCorrect": true, + "iou": 0.7427838540217232, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": "1.00", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0.48466749781196994, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0.4424261325415084, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "2.818", + "predicted": "2.818", + "valueCorrect": true, + "iou": 0.35111142550542873, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31.000", + "predicted": "31.000", + "valueCorrect": true, + "iou": 0.30163871036039036, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 12465, + "outputTokens": 778, + "modelCalls": 1, + "costUSD": 0.085665 + } + }, + { + "docId": "cord/test/6", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "EGG TART", + "predicted": "EGG TART", + "valueCorrect": true, + "iou": 0.7158504259726742, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7649286484588567, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13000", + "valueCorrect": true, + "iou": 0.866392819472924, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "CHOCO CUS ARD PASTRY", + "predicted": "CHOCO CUSTARD PASTRY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.7675085853927492, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24000", + "valueCorrect": true, + "iou": 0.8313469144968956, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "REDBEAN BREAD", + "predicted": "REDBEAN BREAD", + "valueCorrect": true, + "iou": 0.8217067809798223, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8397910337738684, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "9,000", + "predicted": "9000", + "valueCorrect": true, + "iou": 0.910210258392983, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "46,000", + "predicted": "46000", + "valueCorrect": true, + "iou": 0.8692076712328761, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 864, + "modelCalls": 1, + "costUSD": 0.03959 + } + }, + { + "docId": "cord/test/7", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 15", + "predicted": "Kupon 15", + "valueCorrect": true, + "iou": 0.5966867182490869, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "100,000", + "predicted": "100000", + "valueCorrect": true, + "iou": 0.4602725853411543, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "ADD CHICKEN BOX", + "predicted": "ADD CHICKEN BOX", + "valueCorrect": true, + "iou": 0.7509587431053154, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "909", + "predicted": "909", + "valueCorrect": true, + "iou": 0.150779204637942, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "100,909", + "predicted": "100909", + "valueCorrect": true, + "iou": 0.3962275770295249, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "10,091", + "predicted": "10091", + "valueCorrect": true, + "iou": 0.42958477062778627, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "111,000", + "predicted": "111000", + "valueCorrect": true, + "iou": 0.45963024616985976, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 814, + "modelCalls": 1, + "costUSD": 0.034365 + } + }, + { + "docId": "cord/test/8", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[REG] BLACK SAKURA", + "predicted": "[REG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0.8343234862530206, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8362362116166174, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "45,455", + "predicted": "45,455", + "valueCorrect": true, + "iou": 0.7159530936671168, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COOKIE DOH SAUCES", + "predicted": "COOKIE DOH SAUCES", + "valueCorrect": true, + "iou": 0.8654711721273758, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8879061912518158, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.48654621764178657, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "NATA DE COCO", + "predicted": "NATA DE COCO", + "valueCorrect": true, + "iou": 0.876299206924634, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7629045527173569, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4970176517983176, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "45,455", + "predicted": "45,455", + "valueCorrect": true, + "iou": 0.8090205389723376, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "4,545", + "predicted": "4,545", + "valueCorrect": true, + "iou": 0.8211804829906545, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "50,000", + "predicted": "50,000", + "valueCorrect": true, + "iou": 0.8788069764963309, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1109, + "modelCalls": 1, + "costUSD": 0.04694 + } + }, + { + "docId": "cord/test/9", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Bumbu Kaldu Ayam 1", + "predicted": "Bumbu Kaldu Ayam", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": "1", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0.25344103975811283, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0.15586646465523998, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0.14963295553766537, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": "0 %", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": null, + "predicted": "36000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4353, + "outputTokens": 1461, + "modelCalls": 2, + "costUSD": 0.065595 + } + }, + { + "docId": "cord/test/10", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "CINNAMON SUGAR", + "predicted": "CINNAMON SUGAR", + "valueCorrect": true, + "iou": 0.8339997671784578, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1 x", + "predicted": "1", + "valueCorrect": true, + "iou": 0.1833580439514428, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "17,000", + "predicted": "17000", + "valueCorrect": true, + "iou": 0.5858383387441223, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17,000", + "predicted": "17000", + "valueCorrect": true, + "iou": 0.8239800159077797, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "17,000", + "predicted": "17000", + "valueCorrect": true, + "iou": 0.8313517203590538, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "17,000", + "predicted": "17000", + "valueCorrect": true, + "iou": 0.8224351733389363, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 621, + "modelCalls": 1, + "costUSD": 0.028575 + } + }, + { + "docId": "cord/test/11", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BANABERRY FRESH CREAM CAK", + "predicted": "BANABERRY FRESH CREAM CAK", + "valueCorrect": true, + "iou": 0.829818279349631, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.675308724733086, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "230,000", + "predicted": "230000", + "valueCorrect": true, + "iou": 0.7662472857238694, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "230,000", + "predicted": "230000", + "valueCorrect": true, + "iou": 0.8928988023466835, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 743, + "modelCalls": 1, + "costUSD": 0.032235 + } + }, + { + "docId": "cord/test/12", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BLUS WANITA", + "predicted": "BLUS WANITA", + "valueCorrect": true, + "iou": 0.7288490853658531, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6697959533607701, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@120,000", + "predicted": "@120,000", + "valueCorrect": true, + "iou": 0.7515038145459476, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": null, + "predicted": "120,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "SHOPPING BAG", + "predicted": "SHOPPING BAG", + "valueCorrect": true, + "iou": 0.6811370960666873, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.659539874806923, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": "@880", + "predicted": "@880", + "valueCorrect": true, + "iou": 0.7439701991054346, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "120,000", + "predicted": "120,000", + "valueCorrect": true, + "iou": 0.7804710341093688, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 1191, + "modelCalls": 1, + "costUSD": 0.045675 + } + }, + { + "docId": "cord/test/14", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "RedVelvet Cheese Slc", + "predicted": "RedVelvet Cheese Slc", + "valueCorrect": true, + "iou": 0.7564484778336535, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5992703999999954, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38000", + "valueCorrect": true, + "iou": 0.7951640547945173, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Chicken Baked Rice", + "predicted": "Chicken Baked Rice", + "valueCorrect": true, + "iou": 0.7823397545847177, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6537495272727253, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "55,000", + "predicted": "55000", + "valueCorrect": true, + "iou": 0.7791540402684567, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Fish n Chips", + "predicted": "Fish n Chips", + "valueCorrect": true, + "iou": 0.7981327058823539, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5287679999999992, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65000", + "valueCorrect": true, + "iou": 0.7791540402684567, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Mushroom Soup", + "predicted": "Mushroom Soup", + "valueCorrect": true, + "iou": 0.6813685832386298, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5768378181818155, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "35,000", + "predicted": "35000", + "valueCorrect": true, + "iou": 0.7166293333333336, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Chamomile", + "predicted": "Chamomile", + "valueCorrect": true, + "iou": 0.7423394594594602, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5738709284026022, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25000", + "valueCorrect": true, + "iou": 0.73096192, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Iced Tea", + "predicted": "Iced Tea", + "valueCorrect": true, + "iou": 0.8617086185566973, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.576635768493352, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "20,000", + "predicted": "20000", + "valueCorrect": true, + "iou": 0.7035997090909063, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "238,000", + "predicted": "238000", + "valueCorrect": true, + "iou": 0.8659976878612716, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,850", + "predicted": "17850", + "valueCorrect": true, + "iou": 0.8280483221476511, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "25,585", + "predicted": "25585", + "valueCorrect": true, + "iou": 0.7615999999999971, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "281,435", + "predicted": "281435", + "valueCorrect": true, + "iou": 0.9319026970814462, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 8505, + "outputTokens": 1867, + "modelCalls": 1, + "costUSD": 0.098535 + } + }, + { + "docId": "cord/test/15", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Lemon Tea (L)", + "predicted": "Lemon Tea (L)", + "valueCorrect": true, + "iou": 0.7976648204002972, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6509130509173026, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25.000", + "predicted": "25.000", + "valueCorrect": true, + "iou": 0.8229088225758201, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Caramel Small", + "predicted": "Caramel Small", + "valueCorrect": true, + "iou": 0.8366718161727005, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7177962594128809, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38.000", + "predicted": "38.000", + "valueCorrect": true, + "iou": 0.8566186518602785, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "63.000", + "predicted": "63.000", + "valueCorrect": true, + "iou": 0.7854185276217978, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 859, + "modelCalls": 1, + "costUSD": 0.035715 + } + }, + { + "docId": "cord/test/16", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[LG] BLACK SAKURA", + "predicted": "[LG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0.8638577907620562, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7145881854753332, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "59,091", + "predicted": "59091", + "valueCorrect": true, + "iou": 0.8323383137425263, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "LONGAN", + "predicted": "LONGAN", + "valueCorrect": true, + "iou": 0.66260003766577, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5743269563317964, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.8206837104704299, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ROASTED ALMOND", + "predicted": "ROASTED ALMOND", + "valueCorrect": true, + "iou": 0.757100507572592, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6917499519624809, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.8081412750673422, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "IVANGGO", + "predicted": "MANGGO", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6507325942149493, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.732060010845332, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "IVINERAL WATER (bundling) 5k", + "predicted": "MINERAL WATER (bundling) 5k", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6095140677468811, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "4,545", + "predicted": "4545", + "valueCorrect": true, + "iou": 0.8959632056905936, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "63,636", + "predicted": "63636", + "valueCorrect": true, + "iou": 0.7044691294239199, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "6,364", + "predicted": "6364", + "valueCorrect": true, + "iou": 0.6564250006718078, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "70,000", + "predicted": "70000", + "valueCorrect": true, + "iou": 0.7654076377630198, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1571, + "modelCalls": 1, + "costUSD": 0.0608 + } + }, + { + "docId": "cord/test/17", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Goblin's Mace", + "predicted": "Goblin's Mace", + "valueCorrect": true, + "iou": 0.8260756192959625, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6438177874186575, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25,000", + "valueCorrect": true, + "iou": 0.7538055029008004, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Mozarella Hot Dog", + "predicted": "Mozarella Hot Dog", + "valueCorrect": true, + "iou": 0.840097061328684, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.6437529029261515, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38,000", + "valueCorrect": true, + "iou": 0.8384168488216095, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Chili Pepper Croquette", + "predicted": "Chili Pepper Croquette", + "valueCorrect": true, + "iou": 0.7393108267458612, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5940616707841494, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14,000", + "valueCorrect": true, + "iou": 0.858896999131967, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cheese Croquette", + "predicted": "Cheese Croquette", + "valueCorrect": true, + "iou": 0.6408621733640606, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7936507936507977, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14,000", + "valueCorrect": true, + "iou": 0.7268005271726484, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Plastik Amook", + "predicted": "Plastik Amook", + "valueCorrect": true, + "iou": 0.8521574034671463, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.551258613415841, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4260273972602728, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Plastik Putih Take Away", + "predicted": "Plastik Putih Take Away", + "valueCorrect": true, + "iou": 0.7021378106884, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7543140299757466, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4501130710085933, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "91,000", + "predicted": "91,000", + "valueCorrect": true, + "iou": 0.8117895163486489, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "(0)", + "predicted": "(0)", + "valueCorrect": true, + "iou": 0.633181126331812, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "Rp. 91,000", + "predicted": "Rp. 91,000", + "valueCorrect": true, + "iou": 0.8347713425701029, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 1774, + "modelCalls": 1, + "costUSD": 0.06745 + } + }, + { + "docId": "cord/test/18", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Propolis Cair", + "predicted": "Propolis Cair", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": "156000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4360, + "outputTokens": 2520, + "modelCalls": 2, + "costUSD": 0.0974 + } + }, + { + "docId": "cord/test/19", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "AMBUSH DBL CHS", + "predicted": "AMBUSH DBL CHS", + "valueCorrect": true, + "iou": 0.7073179849291227, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "4", + "predicted": "4", + "valueCorrect": true, + "iou": 0.5119811342709301, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0.7274057142857159, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AMBUSH CHS BUR", + "predicted": "AMBUSH CHS BUR", + "valueCorrect": true, + "iou": 0.7389005925317126, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "10", + "predicted": "10", + "valueCorrect": true, + "iou": 0.6511691272067354, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "100000", + "predicted": "100000", + "valueCorrect": true, + "iou": 0.5247397098056842, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "160000", + "predicted": "160000", + "valueCorrect": true, + "iou": 0.7392822695860964, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "16000", + "predicted": "16000", + "valueCorrect": true, + "iou": 0.6600044535305944, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "176000", + "predicted": "176000", + "valueCorrect": true, + "iou": 0.7034192932102685, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 786, + "modelCalls": 1, + "costUSD": 0.033525 + } + }, + { + "docId": "cord/test/20", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "cashew nuts chkn", + "predicted": "cashew nuts chkn", + "valueCorrect": true, + "iou": 0.8246695158014271, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.4121059959043924, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "64,500", + "predicted": "93500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "garlic pepper beef", + "predicted": "garlic pepper beef", + "valueCorrect": true, + "iou": 0.8464176000000014, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.44087432698576345, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "79,500", + "predicted": "72500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "red curry beef", + "predicted": "red curry beef", + "valueCorrect": true, + "iou": 0.697603925277195, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.38605218558878474, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "69,500", + "predicted": "69500", + "valueCorrect": true, + "iou": 0.4606840334616641, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "phad thai", + "predicted": "phad thai", + "valueCorrect": true, + "iou": 0.6459253828387005, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.4146240034084665, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "64,500", + "predicted": "64500", + "valueCorrect": true, + "iou": 0.379463696322686, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "steamed rice", + "predicted": "steamed rice", + "valueCorrect": true, + "iou": 0.48836971539253654, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "4", + "predicted": "4", + "valueCorrect": true, + "iou": 0.3289367720448421, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "47,600", + "predicted": "44000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "325,600", + "predicted": "344000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,908", + "predicted": "17200", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "34,351", + "predicted": "36120", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "377,859", + "predicted": "397320", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1572, + "modelCalls": 1, + "costUSD": 0.06083 + } + }, + { + "docId": "cord/test/29", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "COLD OCHA", + "predicted": "COLD OCHA", + "valueCorrect": true, + "iou": 0.6993446723771161, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7979057090417228, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "10,909", + "predicted": "10909", + "valueCorrect": true, + "iou": 0.6953331035710916, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BCA PROMO", + "predicted": "BCA PROMO", + "valueCorrect": true, + "iou": 0.9120423534641415, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8007592190445528, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "54,455", + "predicted": "54455", + "valueCorrect": true, + "iou": 0.6972702612124302, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ABURA UDON SPICY", + "predicted": "|-ABURA UDON SPICY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7970067485753457, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.6696183244769551, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": ":-KAKE UDON", + "predicted": "|-KAKE UDON", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7800956962681426, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.6150679874001386, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "SKEWERED SHUMAI", + "predicted": "SKEWERED SHUMAI", + "valueCorrect": true, + "iou": 0.8201986076415521, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7835211860680217, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12727", + "valueCorrect": true, + "iou": 0.599830524074147, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "SKEWERED TOFU ROLL", + "predicted": "SKEWERED TOFU ROLL", + "valueCorrect": true, + "iou": 0.7947076375475453, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7535473804130487, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12727", + "valueCorrect": true, + "iou": 0.6099080381586052, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "EBI TEMPURA", + "predicted": "EBI TEMPURA", + "valueCorrect": true, + "iou": 0.8624769538740162, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6932400067736995, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "13,636", + "predicted": "13636", + "valueCorrect": true, + "iou": 0.6572060412129339, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "KITSUNE UDON (KIDS)", + "predicted": "KITSUNE UDON (KIDS)", + "valueCorrect": true, + "iou": 0.7815648778420597, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8524675770046133, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "28,182", + "predicted": "28182", + "valueCorrect": true, + "iou": 0.7030761914026663, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "132,636", + "predicted": "132636", + "valueCorrect": true, + "iou": 0.8569190350705842, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "13,264", + "predicted": "13264", + "valueCorrect": true, + "iou": 0.9002512235517395, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "145,900", + "predicted": "145900", + "valueCorrect": true, + "iou": 0.9040016769194672, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 2344, + "modelCalls": 1, + "costUSD": 0.08399 + } + }, + { + "docId": "cord/test/30", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 3", + "predicted": "Kupon 3", + "valueCorrect": true, + "iou": 0.8303601599999967, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28,636", + "predicted": "28,636", + "valueCorrect": true, + "iou": 0.8719757482900103, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,636", + "predicted": "28,636", + "valueCorrect": true, + "iou": 0.7819400930232572, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "2,864", + "predicted": "2,864", + "valueCorrect": true, + "iou": 0.8171104864864865, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31,500", + "predicted": "31,500", + "valueCorrect": true, + "iou": 0.6523331306990879, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1312, + "modelCalls": 1, + "costUSD": 0.05303 + } + }, + { + "docId": "cord/test/35", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TWIST DONUT", + "predicted": "TWIST DONUT", + "valueCorrect": true, + "iou": 0.7835529186602872, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.8023739471537845, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "18,000", + "predicted": "18000", + "valueCorrect": true, + "iou": 0.7571319005071238, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BANANA DONUT", + "predicted": "BANANA DONUT", + "valueCorrect": true, + "iou": 0.8016743034055714, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8104180639391892, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11000", + "valueCorrect": true, + "iou": 0.8592784189091514, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CREAMCHEESE BREAD", + "predicted": "CREAMCHEESE BREAD", + "valueCorrect": true, + "iou": 0.9070271556412157, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7244769391847535, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13000", + "valueCorrect": true, + "iou": 0.7634755953598302, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "FRANKFRUT SAUSAGE ROLL", + "predicted": "FRANKFRUT SAUSAGE ROLL", + "valueCorrect": true, + "iou": 0.836711571244438, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5862996055308917, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "12,000", + "predicted": "12000", + "valueCorrect": true, + "iou": 0.7671633664257212, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "54,000", + "predicted": "54000", + "valueCorrect": true, + "iou": 0.9230838606513019, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1379, + "modelCalls": 1, + "costUSD": 0.05504 + } + }, + { + "docId": "cord/test/38", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "XLB Org Pork 10x", + "predicted": "XLO Org Pork 10x", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6655643918850811, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "78,000", + "predicted": "78000", + "valueCorrect": true, + "iou": 0.5763023069559248, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Sicito Babi", + "predicted": "Sicito Babi", + "valueCorrect": true, + "iou": 0.7493273744358183, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6766301921302466, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65000", + "valueCorrect": true, + "iou": 0.7033577079927786, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Hotplate Tahu", + "predicted": "Hotplate Tahu", + "valueCorrect": true, + "iou": 0.7611177328214528, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6114028635405155, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "58,000", + "predicted": "58000", + "valueCorrect": true, + "iou": 0.764186226021561, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "DF Fish Fillet Garlc", + "predicted": "DF Fish Fillet Garlic", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6814462120675362, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "108,000", + "predicted": "108000", + "valueCorrect": true, + "iou": 0.7452628553256605, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "LM Pork Belly", + "predicted": "LM Pork Belly", + "valueCorrect": true, + "iou": 0.6038410326450785, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6392904840727034, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "88,000", + "predicted": "83000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Siew Mai", + "predicted": "Siew Mai", + "valueCorrect": true, + "iou": 0.6581486460396626, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.532394583813751, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "38,800", + "predicted": "38800", + "valueCorrect": true, + "iou": 0.7636908742125446, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Kwan Yin Cup", + "predicted": "Kwan Yin Cup", + "valueCorrect": true, + "iou": 0.645604287795534, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "5", + "predicted": "5", + "valueCorrect": true, + "iou": 0.5207966272073375, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "50,000", + "predicted": "50000", + "valueCorrect": true, + "iou": 0.7977799882523833, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Herbal Jelly", + "predicted": "Herbal Jelly", + "valueCorrect": true, + "iou": 0.654145226319082, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.38911754676550375, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "38,000", + "predicted": "33000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Onde-Onde", + "predicted": "Onde-Onde", + "valueCorrect": true, + "iou": 0.7526116264110794, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.33485840776671816, + "predictedPage": 0 + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38000", + "valueCorrect": true, + "iou": 0.7248547360135673, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "561,800", + "predicted": "561800", + "valueCorrect": true, + "iou": 0.5502543446278729, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "42,135", + "predicted": "42135", + "valueCorrect": true, + "iou": 0.6344363893340013, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "60,394", + "predicted": "60394", + "valueCorrect": true, + "iou": 0.6147234670474858, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "664,329", + "predicted": "664329", + "valueCorrect": true, + "iou": 0.5757795563727953, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 12465, + "outputTokens": 2382, + "modelCalls": 1, + "costUSD": 0.133785 + } + }, + { + "docId": "cord/test/41", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0.8856100353197268, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0.7336423283503785, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Cappucino (Hot/Iced)", + "predicted": "Cappucino (Hot/Iced)", + "valueCorrect": true, + "iou": 0.8797579457800513, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.7484050359846025, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Naga Lychee Juice", + "predicted": "Naga Lychee Juice", + "valueCorrect": true, + "iou": 0.7902969663189356, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "34632.00", + "predicted": "34632.00", + "valueCorrect": true, + "iou": 0.6323527398323064, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cafe Latte", + "predicted": "Cafe Latte", + "valueCorrect": true, + "iou": 0.7080725074092009, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.4687506034500339, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Hot Chocolate Marshmallo", + "predicted": "Hot Chocolate Marshmallo", + "valueCorrect": true, + "iou": 0.5222534517376489, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.2911891829841219, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Chicken Cesar Salad", + "predicted": "Chicken Cesar Salad", + "valueCorrect": true, + "iou": 0.44402011800741004, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "43290.00", + "predicted": "43290.00", + "valueCorrect": true, + "iou": 0.15033896323933046, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0.37146160169469766, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0.08758858059296559, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0.3640193697729364, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0.06496934482548963, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Lychee/Peach/Lemon/Mango", + "predicted": "Lychee/Peach/Lemon/Mango", + "valueCorrect": true, + "iou": 0.2522961984412319, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "25974.00", + "predicted": "25974.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.9.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0.23125003786524642, + "predictedPage": 0 + }, + { + "path": "lineItems.9.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0.018249925363241125, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "316017.00", + "predicted": "316017.00", + "valueCorrect": true, + "iou": 0.6632756697163117, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "48982.68", + "predicted": "48982.68", + "valueCorrect": true, + "iou": 0.5115471729442655, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "365000.00", + "predicted": "365000.00", + "valueCorrect": true, + "iou": 0.7190512941176486, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2313, + "outputTokens": 2230, + "modelCalls": 1, + "costUSD": 0.078465 + } + } + ] + }, + { + "model": "gpt-5.6-luna", + "docs": [ + { + "docId": "cord/test/0", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "-TICKET CP", + "predicted": "-TICKET CP", + "valueCorrect": true, + "iou": 0.3249336322036175, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.5281881217968074, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "60000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60.000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "60.000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0.4580770909090929, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "-60.000", + "predicted": "-60000", + "valueCorrect": true, + "iou": 0.45018947368421214, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "5.455", + "predicted": "5455", + "valueCorrect": true, + "iou": 0.5332114285714291, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "60.000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0.3824160586351491, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1758, + "outputTokens": 1349, + "modelCalls": 1, + "costUSD": 0.009852 + } + }, + { + "docId": "cord/test/1", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "J.STB PROMO", + "predicted": "J. STB PROMO", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17500", + "predicted": "17500", + "valueCorrect": true, + "iou": 0.9385833528286402, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Y.B.BAT", + "predicted": "Y.B.BAT", + "valueCorrect": true, + "iou": 0.788690529154029, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "46000", + "predicted": "46000", + "valueCorrect": true, + "iou": 0.898745709368988, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Y.BASO PROM", + "predicted": "Y.BASO PROM", + "valueCorrect": true, + "iou": 0.809653028927722, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "27500", + "predicted": "27500", + "valueCorrect": true, + "iou": 0.815061521298339, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "91000", + "predicted": "91000", + "valueCorrect": true, + "iou": 0.7865448929452407, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 646, + "modelCalls": 1, + "costUSD": 0.006722 + } + }, + { + "docId": "cord/test/2", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "JASMINE MT ( L )", + "predicted": "SUNSHINE MT ( L )", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.43673551103705555, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24.000", + "valueCorrect": true, + "iou": 0.7867321146402637, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COCONUT JELLY ( L )", + "predicted": "COCONUT JELLY ( L )", + "valueCorrect": true, + "iou": 0.6962409412242491, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "4,000", + "predicted": "4.000", + "valueCorrect": true, + "iou": 0.7676623728609945, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,000", + "predicted": "28.000", + "valueCorrect": true, + "iou": 0.7828501895119248, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "28,000", + "predicted": "28.000", + "valueCorrect": true, + "iou": 0.5176860916270122, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 5917, + "outputTokens": 1915, + "modelCalls": 2, + "costUSD": 0.017407 + } + }, + { + "docId": "cord/test/3", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "DONAT GULA", + "predicted": "DONAT GULA", + "valueCorrect": true, + "iou": 0.4655601588024114, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1X", + "predicted": "1X", + "valueCorrect": true, + "iou": 0.523147493849749, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@11000", + "predicted": "@11000", + "valueCorrect": true, + "iou": 0.5650820704484878, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0.46685305026348844, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0.5009331562638344, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0.11277656157427192, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 739, + "modelCalls": 1, + "costUSD": 0.007168 + } + }, + { + "docId": "cord/test/4", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "ICE BLACKCOFFE", + "predicted": "ICE BLACKCOFFE", + "valueCorrect": true, + "iou": 0.7140098234613828, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "1", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "82,000", + "predicted": "82,000", + "valueCorrect": true, + "iou": 0.2895963073035434, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AVOCADO COFFEE", + "predicted": "AVOCADO COFFEE", + "valueCorrect": true, + "iou": 0.6910825377481531, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.4394756367085779, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "61,000", + "predicted": "62,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CHIKEN KATSU FF", + "predicted": "CHICKEN KATSU PF", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.24098634378005268, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "51,000", + "predicted": "50,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "194,000", + "predicted": "194,000", + "valueCorrect": true, + "iou": 0.6270509361427453, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "19,400", + "predicted": "19,400", + "valueCorrect": true, + "iou": 0.6709602264819453, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "174,600", + "predicted": "174,600", + "valueCorrect": true, + "iou": 0.4607371585486942, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1524, + "modelCalls": 1, + "costUSD": 0.011878 + } + }, + { + "docId": "cord/test/5", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TRAD KY TOAST CARTE", + "predicted": "TRAD KY TOAST CARTE", + "valueCorrect": true, + "iou": 0.47709361760249236, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": "1.00", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0.6110613447383183, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "2.818", + "predicted": "2.818", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31.000", + "predicted": "31.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 12465, + "outputTokens": 1481, + "modelCalls": 1, + "costUSD": 0.021351 + } + }, + { + "docId": "cord/test/6", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "EGG TART", + "predicted": "EGG TART", + "valueCorrect": true, + "iou": 0.6514932131756087, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7156052247891889, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13000", + "valueCorrect": true, + "iou": 0.8076202369337975, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "CHOCO CUS ARD PASTRY", + "predicted": "CHOCO CUSTARD PASTRY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.5008832649314465, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24000", + "valueCorrect": true, + "iou": 0.6618253522867104, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "REDBEAN BREAD", + "predicted": "REDBEAN BREAD", + "valueCorrect": true, + "iou": 0.41478638893583075, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5816294715545735, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "9,000", + "predicted": "9000", + "valueCorrect": true, + "iou": 0.4935754227653358, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "46,000", + "predicted": "46000", + "valueCorrect": true, + "iou": 0.8036835317739442, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 914, + "modelCalls": 1, + "costUSD": 0.008218 + } + }, + { + "docId": "cord/test/7", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 15", + "predicted": "Kucan 15", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "100,000", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "ADD CHICKEN BOX", + "predicted": "ADD CHICKEN BOX", + "valueCorrect": true, + "iou": 0.6237488925243464, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "909", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "subtotal", + "compare": "money", + "expected": "100,909", + "predicted": "", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "10,091", + "predicted": "", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "111,000", + "predicted": "", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 2027, + "modelCalls": 1, + "costUSD": 0.014151 + } + }, + { + "docId": "cord/test/8", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[REG] BLACK SAKURA", + "predicted": "[REG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0.8819958221776065, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6500962461059283, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "45,455", + "predicted": "45455", + "valueCorrect": true, + "iou": 0.7785820584822939, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COOKIE DOH SAUCES", + "predicted": "COOKIE DOH SAUCES", + "valueCorrect": true, + "iou": 0.8789719066759314, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7687720155097078, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.7273802208442702, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "NATA DE COCO", + "predicted": "NATA DE COCO", + "valueCorrect": true, + "iou": 0.7772993721368372, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5965077640129627, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.6776593821156116, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "45,455", + "predicted": "45455", + "valueCorrect": true, + "iou": 0.6103766970486051, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "4,545", + "predicted": "4545", + "valueCorrect": true, + "iou": 0.6409863556427623, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "50,000", + "predicted": "50000", + "valueCorrect": true, + "iou": 0.865163176705018, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 944, + "modelCalls": 1, + "costUSD": 0.008398 + } + }, + { + "docId": "cord/test/9", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Bumbu Kaldu Ayam 1", + "predicted": "Bumbu Kaldu Ayam 1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "36000", + "predicted": "30000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": null, + "predicted": "36000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4343, + "outputTokens": 1857, + "modelCalls": 2, + "costUSD": 0.015485 + } + }, + { + "docId": "cord/test/10", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "CINNAMON SUGAR", + "predicted": "CINNAMON SUGAR", + "valueCorrect": true, + "iou": 0.693120483472298, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1 x", + "predicted": "1", + "valueCorrect": true, + "iou": 0.14774400000000035, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0.7485120000000005, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0.655828469635629, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0.6541956125897659, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0.4039814371420423, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 570, + "modelCalls": 1, + "costUSD": 0.005409 + } + }, + { + "docId": "cord/test/11", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BANABERRY FRESH CREAM CAK", + "predicted": "BANABERRY FRESH CREAM CAK", + "valueCorrect": true, + "iou": 0.49109158057357294, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6379009079626364, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "230,000", + "predicted": "230,000", + "valueCorrect": true, + "iou": 0.3284885037222402, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "230,000", + "predicted": "230,000", + "valueCorrect": true, + "iou": 0.6996086479338842, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 814, + "modelCalls": 1, + "costUSD": 0.006873 + } + }, + { + "docId": "cord/test/12", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BLUS WANITA", + "predicted": "0571-1854 BLUS WANITA", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6868983183902834, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@120,000", + "predicted": "120000", + "valueCorrect": true, + "iou": 0.3822304147465428, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": null, + "predicted": "120000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "SHOPPING BAG", + "predicted": "1002-0000 SHOPPING BAG", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.42427727451752884, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": "@880", + "predicted": "880", + "valueCorrect": true, + "iou": 0.25374999999999903, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "120,000", + "predicted": "120000", + "valueCorrect": true, + "iou": 0.587899504358231, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 971, + "modelCalls": 1, + "costUSD": 0.007815 + } + }, + { + "docId": "cord/test/14", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "RedVelvet Cheese Slc", + "predicted": "RedVelvet Chees Slic", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5951466629857103, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "38,000", + "predicted": "30,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Chicken Baked Rice", + "predicted": "Chicken Baked Rice", + "valueCorrect": true, + "iou": 0.7756077848525281, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5331005515446485, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "55,000", + "predicted": "55,000", + "valueCorrect": true, + "iou": 0.7416454899400838, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Fish n Chips", + "predicted": "Fish n Chips", + "valueCorrect": true, + "iou": 0.517471733392686, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.3833518930957679, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65,000", + "valueCorrect": true, + "iou": 0.609342685785192, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Mushroom Soup", + "predicted": "Mushroom Soup", + "valueCorrect": true, + "iou": 0.4750912942427476, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.25510600006604256, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "35,000", + "predicted": "35,000", + "valueCorrect": true, + "iou": 0.4423195558297336, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Chamomile", + "predicted": "Chamomile", + "valueCorrect": true, + "iou": 0.2695615333674253, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.15104867271313405, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25,000", + "valueCorrect": true, + "iou": 0.2972326358849473, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Iced Tea", + "predicted": "Iced Tea", + "valueCorrect": true, + "iou": 0.16675262917372835, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.11202112313303725, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "20,000", + "predicted": "20,000", + "valueCorrect": true, + "iou": 0.232143528085667, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "238,000", + "predicted": "230,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,850", + "predicted": "17,850", + "valueCorrect": true, + "iou": 0.27373581114216405, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "25,585", + "predicted": "25,585", + "valueCorrect": true, + "iou": 0.23364408315337373, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "281,435", + "predicted": "281,435", + "valueCorrect": true, + "iou": 0.4295747115388498, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 8505, + "outputTokens": 1318, + "modelCalls": 1, + "costUSD": 0.016413 + } + }, + { + "docId": "cord/test/15", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Lemon Tea (L)", + "predicted": "Lemon Tea (L)", + "valueCorrect": true, + "iou": 0.8856123471453409, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5646359630769248, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25.000", + "predicted": "25. 000", + "valueCorrect": true, + "iou": 0.3022930919796989, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Caramel Small", + "predicted": "Caramel Small", + "valueCorrect": true, + "iou": 0.8156222553138096, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7259605239560433, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38.000", + "predicted": "38. 000", + "valueCorrect": true, + "iou": 0.4003861638393951, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "63.000", + "predicted": "63. 000", + "valueCorrect": true, + "iou": 0.8043948754156321, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 591, + "modelCalls": 1, + "costUSD": 0.005535 + } + }, + { + "docId": "cord/test/16", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[LG] BLACK SAKURA", + "predicted": "[LG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0.8625209780893547, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.698359908018018, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "59,091", + "predicted": "59091", + "valueCorrect": true, + "iou": 0.7508140367923128, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "LONGAN", + "predicted": "LONGAN", + "valueCorrect": true, + "iou": 0.60202644998717, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7646475029241805, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4473420169707748, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ROASTED ALMOND", + "predicted": "ROASTED ALMOND", + "valueCorrect": true, + "iou": 0.5177989152356086, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8498057142857095, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.47423703098558495, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "IVANGGO", + "predicted": "MANGO", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6068084635800659, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4242735062773248, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "IVINERAL WATER (bundling) 5k", + "predicted": "MINERAL WATER (bundling) 5k", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5299593141155654, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "4,545", + "predicted": "4545", + "valueCorrect": true, + "iou": 0.6311486846965113, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "63,636", + "predicted": "63636", + "valueCorrect": true, + "iou": 0.5762201027055761, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "6,364", + "predicted": "6364", + "valueCorrect": true, + "iou": 0.4381208457251694, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "70,000", + "predicted": "70000", + "valueCorrect": true, + "iou": 0.5583083899230551, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1339, + "modelCalls": 1, + "costUSD": 0.010768 + } + }, + { + "docId": "cord/test/17", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Goblin's Mace", + "predicted": "Goblin's Mace", + "valueCorrect": true, + "iou": 0.814429595629954, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7404685346807562, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25000", + "valueCorrect": true, + "iou": 0.8342894787406335, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Mozarella Hot Dog", + "predicted": "Mozarella Hot Dog", + "valueCorrect": true, + "iou": 0.8133259790402653, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.6638457921955829, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38000", + "valueCorrect": true, + "iou": 0.865527383620652, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Chili Pepper Croquette", + "predicted": "Chilli Pepper Croquette", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5940616707841494, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14000", + "valueCorrect": true, + "iou": 0.9066122600389515, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cheese Croquette", + "predicted": "Cheese Croquette", + "valueCorrect": true, + "iou": 0.6267378846872489, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7283321194464724, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14000", + "valueCorrect": true, + "iou": 0.8605883627287848, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Plastik Amook", + "predicted": "Plastic Amook", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.551258613415841, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.3890201528839462, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Plastik Putih Take Away", + "predicted": "Plastic Puth Take Away", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5434031775270547, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.3540421932004441, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "91,000", + "predicted": "91000", + "valueCorrect": true, + "iou": 0.7589925162951674, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "(0)", + "predicted": "(0)", + "valueCorrect": true, + "iou": 0.47757699904894335, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "Rp. 91,000", + "predicted": "91000", + "valueCorrect": true, + "iou": 0.7497871959141645, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 1960, + "modelCalls": 1, + "costUSD": 0.014606 + } + }, + { + "docId": "cord/test/18", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Propolis Cair", + "predicted": "Propolis Cair 1", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "156000", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "156000", + "predicted": "150000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": "150000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "156000", + "predicted": "150000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4313, + "outputTokens": 1532, + "modelCalls": 2, + "costUSD": 0.013505 + } + }, + { + "docId": "cord/test/19", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "AMBUSH DBL CHS", + "predicted": "ANGUSH DOL CHS", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "4", + "predicted": "4", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AMBUSH CHS BUR", + "predicted": "ANGUSH CHS BUR", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "10", + "predicted": "10", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "100000", + "predicted": "100000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "160000", + "predicted": "160000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "16000", + "predicted": "16000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "176000", + "predicted": "176000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 1590, + "modelCalls": 1, + "costUSD": 0.011529 + } + }, + { + "docId": "cord/test/20", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "cashew nuts chkn", + "predicted": "cashew nuts chk", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7528130222845806, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "64,500", + "predicted": "64500", + "valueCorrect": true, + "iou": 0.5221901723320233, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "garlic pepper beef", + "predicted": "garlic pepper beef", + "valueCorrect": true, + "iou": 0.7371544019948686, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7871085080390137, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "79,500", + "predicted": "79500", + "valueCorrect": true, + "iou": 0.4535177518843635, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "red curry beef", + "predicted": "red curry beef", + "valueCorrect": true, + "iou": 0.674884690608603, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7871085080390137, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "69,500", + "predicted": "69500", + "valueCorrect": true, + "iou": 0.45097598083646107, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "phad thai", + "predicted": "fried thai", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.8373754791648593, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "64,500", + "predicted": "69500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "steamed rice", + "predicted": "steamed rice", + "valueCorrect": true, + "iou": 0.6329055143628208, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "4", + "predicted": "2", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "47,600", + "predicted": "17500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "325,600", + "predicted": "300500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,908", + "predicted": "17500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "34,351", + "predicted": "59331", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "377,859", + "predicted": "377331", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1668, + "modelCalls": 1, + "costUSD": 0.012742 + } + }, + { + "docId": "cord/test/29", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "COLD OCHA", + "predicted": "COLD OCHA", + "valueCorrect": true, + "iou": 0.8465266935786854, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7114556236697993, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "10,909", + "predicted": "10909", + "valueCorrect": true, + "iou": 0.7804270166985116, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BCA PROMO", + "predicted": "BCA PROMO", + "valueCorrect": true, + "iou": 0.7372602042118455, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.5577969254248574, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "54,455", + "predicted": "54455", + "valueCorrect": true, + "iou": 0.6537623128396365, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ABURA UDON SPICY", + "predicted": "1-ABURA UDON SPICY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.4790294216943428, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.5424502476031207, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": ":-KAKE UDON", + "predicted": "1-KAKKE UDON", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.3542920443568163, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0.4475686841042141, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "SKEWERED SHUMAI", + "predicted": "SKEWERED SHUMAI", + "valueCorrect": true, + "iou": 0.415859366970261, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.2631452077326482, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12727", + "valueCorrect": true, + "iou": 0.5144266799102379, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "SKEWERED TOFU ROLL", + "predicted": "SKEWERED TOFU ROLL", + "valueCorrect": true, + "iou": 0.3694483072390179, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.19166338943729763, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12727", + "valueCorrect": true, + "iou": 0.4067510237113194, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "EBI TEMPURA", + "predicted": "EBI TEMPURA", + "valueCorrect": true, + "iou": 0.3163788156912349, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.19277765448146894, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "13,636", + "predicted": "13636", + "valueCorrect": true, + "iou": 0.34168339916229873, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "KITSUNE UDON (KIDS)", + "predicted": "KITSUNE UDON (KIDS)", + "valueCorrect": true, + "iou": 0.34424717938573113, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.10900070030718195, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "28,182", + "predicted": "28182", + "valueCorrect": true, + "iou": 0.2949830301265597, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "132,636", + "predicted": "132636", + "valueCorrect": true, + "iou": 0.7709593768438596, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "13,264", + "predicted": "13264", + "valueCorrect": true, + "iou": 0.819375628655512, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "145,900", + "predicted": "145900", + "valueCorrect": true, + "iou": 0.9246106655611354, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 2184, + "modelCalls": 1, + "costUSD": 0.015838 + } + }, + { + "docId": "cord/test/30", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 3", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28,636", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,636", + "predicted": "28,636", + "valueCorrect": true, + "iou": 0.6727100684036035, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "2,864", + "predicted": "2,864", + "valueCorrect": true, + "iou": 0.6762956676072077, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31,500", + "predicted": "31,500", + "valueCorrect": true, + "iou": 0.6134767659574467, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 942, + "modelCalls": 1, + "costUSD": 0.008386 + } + }, + { + "docId": "cord/test/35", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TWIST DONUT", + "predicted": "TWIST DONUT", + "valueCorrect": true, + "iou": 0.7391765848939517, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0.636218181818182, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "18,000", + "predicted": "18000", + "valueCorrect": true, + "iou": 0.6246338179183771, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BANANA DONUT", + "predicted": "BANANA DONUT", + "valueCorrect": true, + "iou": 0.7512338229102152, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7927427358538399, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11000", + "valueCorrect": true, + "iou": 0.7185023999999988, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CREAMCHEESE BREAD", + "predicted": "CREAMCHEESE BREAD", + "valueCorrect": true, + "iou": 0.7154099466485335, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.7927427358538418, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13000", + "valueCorrect": true, + "iou": 0.7511615999999999, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "FRANKFRUT SAUSAGE ROLL", + "predicted": "FRANKFRUT SAUSAGE ROLL", + "valueCorrect": true, + "iou": 0.8038787177801668, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.6257949230041508, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "12,000", + "predicted": "12000", + "valueCorrect": true, + "iou": 0.7025154423621971, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "54,000", + "predicted": "54000", + "valueCorrect": true, + "iou": 0.8146658693658108, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1183, + "modelCalls": 1, + "costUSD": 0.009832 + } + }, + { + "docId": "cord/test/38", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "XLB Org Pork 10x", + "predicted": "XLB Org Pork 10x", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "78,000", + "predicted": "78000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Sicito Babi", + "predicted": "Sicito Gabi", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Hotplate Tahu", + "predicted": "Hotplate Tahu", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "58,000", + "predicted": "58000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "DF Fish Fillet Garlc", + "predicted": "DF Fish Fillet Garlic", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "108,000", + "predicted": "108000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "LM Pork Belly", + "predicted": "LM Pork Belly", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "88,000", + "predicted": "88000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Siew Mai", + "predicted": "Siew Mai", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "38,800", + "predicted": "38000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Kwan Yin Cup", + "predicted": "Kwan Yin Cup", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "5", + "predicted": "5", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "50,000", + "predicted": "50000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Herbal Jelly", + "predicted": "Herbal Jelly", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Onde-Onde", + "predicted": "Onde-Onde", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "561,800", + "predicted": "561800", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "42,135", + "predicted": "42135", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "60,394", + "predicted": "60394", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "664,329", + "predicted": "664329", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 12465, + "outputTokens": 1877, + "modelCalls": 1, + "costUSD": 0.023727 + } + }, + { + "docId": "cord/test/41", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0.8138153687560554, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0.39156207752657146, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Cappucino (Hot/Iced)", + "predicted": "Cappucino (Hot/Iced)", + "valueCorrect": true, + "iou": 0.6891542928212805, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.3866609351343727, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Naga Lychee Juice", + "predicted": "Naga Lychee Juice", + "valueCorrect": true, + "iou": 0.599668562671137, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "34632.00", + "predicted": "34362.00", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cafe Latte", + "predicted": "Cafe Latte", + "valueCorrect": true, + "iou": 0.5654052198260496, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.2796055574462964, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Hot Chocolate Marshmallo", + "predicted": "Hot Chocolate Marshmallo", + "valueCorrect": true, + "iou": 0.42456851864609935, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0.195997241888569, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Chicken Cesar Salad", + "predicted": "Chicken Cesar Salad", + "valueCorrect": true, + "iou": 0.4073422935441368, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "43290.00", + "predicted": "43290.00", + "valueCorrect": true, + "iou": 0.07927752794252188, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0.3942237697121774, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0.054959429815004346, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0.4902608822452469, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0.06736675533690044, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Lychee/Peach/Lemon/Mango", + "predicted": "Lychee/Peach/Lemon/Mango", + "valueCorrect": true, + "iou": 0.3662496476052499, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "25974.00", + "predicted": "25974.00", + "valueCorrect": true, + "iou": 0.049687674487175426, + "predictedPage": 0 + }, + { + "path": "lineItems.9.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0.42876996377522664, + "predictedPage": 0 + }, + { + "path": "lineItems.9.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0.10451481225117876, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "316017.00", + "predicted": "316017.60", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "48982.68", + "predicted": "48982.68", + "valueCorrect": true, + "iou": 0.2577868474219766, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "365000.00", + "predicted": "365000.00", + "valueCorrect": true, + "iou": 0.6580207895603319, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2313, + "outputTokens": 1972, + "modelCalls": 1, + "costUSD": 0.014145 + } + } + ] + }, + { + "model": "gpt-5.4-mini", + "docs": [ + { + "docId": "cord/test/0", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "-TICKET CP", + "predicted": "-TICKET CP", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "60.000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60.000", + "predicted": "60.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "60.000", + "predicted": "60.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "-60.000", + "predicted": "-60.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "5.455", + "predicted": "5.455", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "60.000", + "predicted": "60.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1758, + "outputTokens": 360, + "modelCalls": 1, + "costUSD": 0.0029385 + } + }, + { + "docId": "cord/test/1", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "J.STB PROMO", + "predicted": "J.STB PROMO", + "valueCorrect": true, + "iou": 0.09318678125560477, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17500", + "predicted": "17500", + "valueCorrect": true, + "iou": 0.11385991894026304, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Y.B.BAT", + "predicted": "Y.B.BAT", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "46000", + "predicted": "46000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Y.BASO PROM", + "predicted": "Y.BASO PROM", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "27500", + "predicted": "27500", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "91000", + "predicted": "91000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 506, + "modelCalls": 1, + "costUSD": 0.0044115 + } + }, + { + "docId": "cord/test/2", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "JASMINE MT ( L )", + "predicted": "JASMINE MT ( L )", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "24,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COCONUT JELLY ( L )", + "predicted": "COCONUT JELLY ( L )", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "4,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "4,000", + "predicted": "4,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,000", + "predicted": "28,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "28,000", + "predicted": "28,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 477, + "modelCalls": 1, + "costUSD": 0.004197 + } + }, + { + "docId": "cord/test/3", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "DONAT GULA", + "predicted": "DONAT GULA", + "valueCorrect": true, + "iou": 0.11910881721426896, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1X", + "predicted": "1X", + "valueCorrect": true, + "iou": 0.07684317719639747, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@11000", + "predicted": "@11000", + "valueCorrect": true, + "iou": 0.074767371862159, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 327, + "modelCalls": 1, + "costUSD": 0.003522 + } + }, + { + "docId": "cord/test/4", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "ICE BLACKCOFFE", + "predicted": "ICE BLACKCOFFE", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "82,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "82,000", + "predicted": "82,000", + "valueCorrect": true, + "iou": 0.023090751682480037, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AVOCADO COFFEE", + "predicted": "AVOCADO COFFEE", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "67,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "61,000", + "predicted": "67,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CHIKEN KATSU FF", + "predicted": "CHIKEN KATSU FF", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": "51,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "51,000", + "predicted": "51,000", + "valueCorrect": true, + "iou": 0.047933287222054935, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "194,000", + "predicted": "194,000", + "valueCorrect": true, + "iou": 0.28178536027328555, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "19,400", + "predicted": "19,400", + "valueCorrect": true, + "iou": 0.37251922447054076, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "174,600", + "predicted": "174,600", + "valueCorrect": true, + "iou": 0.7446259567586665, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 637, + "modelCalls": 1, + "costUSD": 0.004917 + } + }, + { + "docId": "cord/test/5", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TRAD KY TOAST CARTE", + "predicted": "TRAD KY TOAST CARTE", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": "1.00", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "28.182", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28.182", + "predicted": "28.182", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "2.818", + "predicted": "2.818", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31.000", + "predicted": "31.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4170, + "outputTokens": 329, + "modelCalls": 1, + "costUSD": 0.004608 + } + }, + { + "docId": "cord/test/6", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "EGG TART", + "predicted": "EGG TART", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "CHOCO CUS ARD PASTRY", + "predicted": "CHOCO CUS'ARD PASTRY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "24,000", + "predicted": "24,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "REDBEAN BREAD", + "predicted": "REDBEAN BREAD", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "9,000", + "predicted": "9,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "46,000", + "predicted": "46,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 541, + "modelCalls": 1, + "costUSD": 0.004485 + } + }, + { + "docId": "cord/test/7", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 15", + "predicted": "Kucon 15", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "100,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "100,000", + "predicted": "909", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "ADD CHICKEN BOX", + "predicted": "ADD CHICKEN BOX", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "909", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "subtotal", + "compare": "money", + "expected": "100,909", + "predicted": "100,909", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "10,091", + "predicted": "10,091", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "111,000", + "predicted": "111,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 423, + "modelCalls": 1, + "costUSD": 0.00339525 + } + }, + { + "docId": "cord/test/8", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[REG] BLACK SAKURA", + "predicted": "[REG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "45,455", + "predicted": "45,455", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "COOKIE DOH SAUCES", + "predicted": "COOKIE DOH SAUCES", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "NATA DE COCO", + "predicted": "NATA DE COCO", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "45,455", + "predicted": "45,455", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "4,545", + "predicted": "4,545", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "50,000", + "predicted": "50,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 579, + "modelCalls": 1, + "costUSD": 0.004656 + } + }, + { + "docId": "cord/test/9", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Bumbu Kaldu Ayam 1", + "predicted": "Bumbu Kaldu Ayam 1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": "1", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "36000", + "predicted": "36000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": null, + "predicted": "50000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 330, + "modelCalls": 1, + "costUSD": 0.00297675 + } + }, + { + "docId": "cord/test/10", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "CINNAMON SUGAR", + "predicted": "CINNAMON SUGAR", + "valueCorrect": true, + "iou": 0.10211733352347199, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1 x", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "17,000", + "predicted": "17,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 325, + "modelCalls": 1, + "costUSD": 0.00295425 + } + }, + { + "docId": "cord/test/11", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BANABERRY FRESH CREAM CAK", + "predicted": "BANABERRY FRESH CREAM CAK", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "230,000", + "predicted": "230,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "230,000", + "predicted": "230,000", + "valueCorrect": true, + "iou": 0.2312104140862536, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 312, + "modelCalls": 1, + "costUSD": 0.00289575 + } + }, + { + "docId": "cord/test/12", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "BLUS WANITA", + "predicted": "BLUS WANITA", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "@120,000", + "predicted": "@120,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": null, + "predicted": "120,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "SHOPPING BAG", + "predicted": "SHOPPING BAG", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": "@880", + "predicted": "@880", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": "120,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "120,000", + "predicted": "120,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 455, + "modelCalls": 1, + "costUSD": 0.00353925 + } + }, + { + "docId": "cord/test/14", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "RedVelvet Cheese Slc", + "predicted": "RedVelvet Cheese Slc", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Chicken Baked Rice", + "predicted": "Chicken Baked Rice", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "55,000", + "predicted": "55,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Fish n Chips", + "predicted": "Fish n Chips", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Mushroom Soup", + "predicted": "Mushroom Soup", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "35,000", + "predicted": "35,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Chamomile", + "predicted": "Chamomile", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Iced Tea", + "predicted": "Iced Tea", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "20,000", + "predicted": "20,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "238,000", + "predicted": "238,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,850", + "predicted": "17,850", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "25,585", + "predicted": "25,585", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "281,435", + "predicted": "281,435", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4170, + "outputTokens": 963, + "modelCalls": 1, + "costUSD": 0.007461 + } + }, + { + "docId": "cord/test/15", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Lemon Tea (L)", + "predicted": "Lemon Tea (L)", + "valueCorrect": true, + "iou": 0.26851290444569725, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.08058098292842518, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "25.000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25.000", + "predicted": "25.000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Caramel Small", + "predicted": "Caramel Small", + "valueCorrect": true, + "iou": 0.4080057964906214, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0.11686794112145228, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "38.000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38.000", + "predicted": "38.000", + "valueCorrect": true, + "iou": 0.06606251471902033, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": "63.000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "63.000", + "predicted": "63.000", + "valueCorrect": true, + "iou": 0.1284747607245716, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 461, + "modelCalls": 1, + "costUSD": 0.00356625 + } + }, + { + "docId": "cord/test/16", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "[LG] BLACK SAKURA", + "predicted": "[LG] BLACK SAKURA", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "59,091", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "59,091", + "predicted": "59,091", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "LONGAN", + "predicted": "LONGAN", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ROASTED ALMOND", + "predicted": "ROASTED ALMOND", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "IVANGGO", + "predicted": "MANGGO", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "IVINERAL WATER (bundling) 5k", + "predicted": "MINERAL WATER (bundling) 5k", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": "4,545", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "4,545", + "predicted": "4,545", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "63,636", + "predicted": "63,636", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "6,364", + "predicted": "6,364", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "70,000", + "predicted": "70,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 917, + "modelCalls": 1, + "costUSD": 0.006177 + } + }, + { + "docId": "cord/test/17", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Goblin's Mace", + "predicted": "Goblin's Mace", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "25,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "25,000", + "predicted": "25,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Mozarella Hot Dog", + "predicted": "Mozarella Hot Dog", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "38,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Chili Pepper Croquette", + "predicted": "Chili Pepper Croquette", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": "14,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cheese Croquette", + "predicted": "Cheese Croquette", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": "14,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "14,000", + "predicted": "14,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Plastik Amook", + "predicted": "Plastik Amook", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Plastik Putih Take Away", + "predicted": "Plastik Putih Take Away", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "91,000", + "predicted": "91,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": "(0)", + "predicted": "(0)", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "Rp. 91,000", + "predicted": "Rp. 91,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2846, + "outputTokens": 1045, + "modelCalls": 1, + "costUSD": 0.006837 + } + }, + { + "docId": "cord/test/18", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Propolis Cair", + "predicted": "Propolis Cair 1", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": "156000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "156000", + "predicted": "156000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 319, + "modelCalls": 1, + "costUSD": 0.00292725 + } + }, + { + "docId": "cord/test/19", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "AMBUSH DBL CHS", + "predicted": "4 ANBUSH DBL CHS", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "4", + "predicted": "4", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "60000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "60000", + "predicted": "60000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "AMBUSH CHS BUR", + "predicted": "10 ANBUSH CHS BUR", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "10", + "predicted": "10", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "100000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "100000", + "predicted": "100000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "160000", + "predicted": "160000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "16000", + "predicted": "16000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "176000", + "predicted": "176000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 1989, + "outputTokens": 475, + "modelCalls": 1, + "costUSD": 0.00362925 + } + }, + { + "docId": "cord/test/20", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "cashew nuts chkn", + "predicted": "1 cashew nuts chkn", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "64,500", + "predicted": "64,500", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "garlic pepper beef", + "predicted": "1 garlic pepper beef", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "79,500", + "predicted": "79,500", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "red curry beef", + "predicted": "1 red curry beef", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "69,500", + "predicted": "69,500", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "phad thai", + "predicted": "1 phad thai", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "64,500", + "predicted": "64,500", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "steamed rice", + "predicted": "4 steamed rice", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "4", + "predicted": "4", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "47,600", + "predicted": "47,600", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "325,600", + "predicted": "335,600", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "17,908", + "predicted": "17,780", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "34,351", + "predicted": "34,551", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "377,859", + "predicted": "377,859", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 853, + "modelCalls": 1, + "costUSD": 0.005889 + } + }, + { + "docId": "cord/test/29", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "COLD OCHA", + "predicted": "COLD OCHA", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": "10,909", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "10,909", + "predicted": "10,909", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BCA PROMO", + "predicted": "BCA PROMO", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": "54,455", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "54,455", + "predicted": "54,455", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "ABURA UDON SPICY", + "predicted": "-ABURA UDON SPICY", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": ":-KAKE UDON", + "predicted": "-KAKE UDON", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": "0", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "0", + "predicted": "0", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "SKEWERED SHUMAI", + "predicted": "SKEWERED SHUMAI", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": "12,727", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12,727", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "SKEWERED TOFU ROLL", + "predicted": "SKEWERED TOFU ROLL", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": "12,727", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "12,727", + "predicted": "12,727", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "EBI TEMPURA", + "predicted": "EBI TEMPURA", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": "13,636", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "13,636", + "predicted": "13,636", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "KITSUNE UDON (KIDS)", + "predicted": "KITSUNE UDON (KIDS)", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": "28,182", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "28,182", + "predicted": "28,182", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "132,636", + "predicted": "132,636", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": "13,264", + "predicted": "13,264", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "145,900", + "predicted": "145,900", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 1342, + "modelCalls": 1, + "costUSD": 0.0080895 + } + }, + { + "docId": "cord/test/30", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Kupon 3", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "28,636", + "predicted": null, + "valueCorrect": false, + "iou": null, + "predictedPage": null + }, + { + "path": "subtotal", + "compare": "money", + "expected": "28,636", + "predicted": "28,636", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": "28,636", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": "2,864", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "2,864", + "predicted": "31,500", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "31,500", + "predicted": "50,000", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 231, + "modelCalls": 1, + "costUSD": 0.00309 + } + }, + { + "docId": "cord/test/35", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "TWIST DONUT", + "predicted": "TWIST DONUT", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "2", + "predicted": "2", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "18,000", + "predicted": "18,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "BANANA DONUT", + "predicted": "BANANA DONUT", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "11,000", + "predicted": "11,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "CREAMCHEESE BREAD", + "predicted": "CREAMCHEESE BREAD", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "13,000", + "predicted": "13,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "FRANKFRUT SAUSAGE ROLL", + "predicted": "FRANKFRUT SAUSAGE ROLL", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "12,000", + "predicted": "12,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "54,000", + "predicted": "54,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2734, + "outputTokens": 685, + "modelCalls": 1, + "costUSD": 0.005133 + } + }, + { + "docId": "cord/test/38", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "XLB Org Pork 10x", + "predicted": "XLB Org Pork 10x", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "78,000", + "predicted": "78,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Sicito Babi", + "predicted": "Sicita Babi", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "65,000", + "predicted": "65,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Hotplate Tahu", + "predicted": "Hotplate Tahu", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "58,000", + "predicted": "58,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "DF Fish Fillet Garlc", + "predicted": "DF Fish Fillet Garlic", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "108,000", + "predicted": "108,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "LM Pork Belly", + "predicted": "LM Pork Belly", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "88,000", + "predicted": "88,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Siew Mai", + "predicted": "Siew Mai", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "38,800", + "predicted": "38,800", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Kwan Yin Cup", + "predicted": "Kwan Yin Cup", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": "5", + "predicted": "5", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "50,000", + "predicted": "50,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Herbal Jelly", + "predicted": "Herbal Jelly", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Onde-Onde", + "predicted": "Onde-Onde", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": "1", + "predicted": "1", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "38,000", + "predicted": "38,000", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "561,800", + "predicted": "561,800", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "42,135", + "predicted": "42,135", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": "60,394", + "predicted": "60,394", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "total", + "compare": "money", + "expected": "664,329", + "predicted": "664,329", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 4170, + "outputTokens": 1378, + "modelCalls": 1, + "costUSD": 0.0093285 + } + }, + { + "docId": "cord/test/41", + "dataset": "cord", + "schema": "receipt", + "error": null, + "fields": [ + { + "path": "lineItems.0.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0.21861629469613889, + "predictedPage": 0 + }, + { + "path": "lineItems.0.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.0.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0.1076076900573512, + "predictedPage": 0 + }, + { + "path": "lineItems.1.description", + "compare": "text", + "expected": "Cappucino (Hot/Iced)", + "predicted": "Cappucino (Hot/Iced)", + "valueCorrect": true, + "iou": 0.035348730240376214, + "predictedPage": 0 + }, + { + "path": "lineItems.1.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.1.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.description", + "compare": "text", + "expected": "Naga Lychee Juice", + "predicted": "Naga Lychee Juice", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.2.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.2.amount", + "compare": "money", + "expected": "34632.00", + "predicted": "34632.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.description", + "compare": "text", + "expected": "Cafe Latte", + "predicted": "Cafe Latte", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.3.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.3.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.4.description", + "compare": "text", + "expected": "Hot Chocolate Marshmallo", + "predicted": "Hot Chocolate Marshallo", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "lineItems.4.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.4.amount", + "compare": "money", + "expected": "30303.00", + "predicted": "30303.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.description", + "compare": "text", + "expected": "Chicken Cesar Salad", + "predicted": "Chicken Cesar Salad", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.5.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.5.amount", + "compare": "money", + "expected": "43290.00", + "predicted": "43290.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.6.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.6.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.description", + "compare": "text", + "expected": "Mineral Water", + "predicted": "Mineral Water", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.7.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.7.amount", + "compare": "money", + "expected": "12987.00", + "predicted": "12987.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.description", + "compare": "text", + "expected": "Lychee/Peach/Lemon/Mango", + "predicted": "Lychee/Peach/Lemon/Mango", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.8.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.8.amount", + "compare": "money", + "expected": "25974.00", + "predicted": "25974.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.9.description", + "compare": "text", + "expected": "Chicken Maryland", + "predicted": "Chicken Maryland", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "lineItems.9.quantity", + "compare": "count", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.unitPrice", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "lineItems.9.amount", + "compare": "money", + "expected": "47619.00", + "predicted": "47619.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "subtotal", + "compare": "money", + "expected": "316017.00", + "predicted": "316017.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + }, + { + "path": "discount", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "serviceCharge", + "compare": "money", + "expected": "48982.68", + "predicted": "49892.68", + "valueCorrect": false, + "iou": null, + "predictedPage": 0 + }, + { + "path": "tax", + "compare": "money", + "expected": null, + "predicted": null, + "valueCorrect": true, + "iou": null, + "predictedPage": null + }, + { + "path": "total", + "compare": "money", + "expected": "365000.00", + "predicted": "365000.00", + "valueCorrect": true, + "iou": 0, + "predictedPage": 0 + } + ], + "extraLineItems": 0, + "usage": { + "inputTokens": 2313, + "outputTokens": 1295, + "modelCalls": 1, + "costUSD": 0.00756225 + } + } + ] + } + ] +} diff --git a/packages/evals/src/models.ts b/packages/evals/src/models.ts index eda9be8..c720b26 100644 --- a/packages/evals/src/models.ts +++ b/packages/evals/src/models.ts @@ -77,24 +77,36 @@ function parseProviders(raw: string): Provider[] { return selected; } +/** + * A provider counts as configured only when its API key is a non-empty string, + * matching the playground's model resolver. A blank `KEY=` line in a `.env` + * therefore reads as unset instead of triggering a doomed run against a provider + * with no credential. + */ +function hasApiKey(env: NodeJS.ProcessEnv, provider: Provider): boolean { + const value = env[CATALOG[provider].apiKeyEnv]; + return value !== undefined && value !== ''; +} + /** * Which providers to benchmark. `EVAL_PROVIDERS` selects them explicitly * (comma-separated, e.g. `openai` or `openai,google`); each named provider - * must have its API key set. When unset, every provider whose API key is - * present runs — so setting only `OPENAI_API_KEY` benchmarks OpenAI alone. + * must have its API key set. When unset (or blank), every provider whose API + * key is present runs — so setting only `OPENAI_API_KEY` benchmarks OpenAI + * alone. An empty key value counts as absent, same as the playground. */ export function selectProviders(env: NodeJS.ProcessEnv = process.env): Provider[] { const raw = env['EVAL_PROVIDERS']; - if (raw !== undefined) { + if (raw !== undefined && raw.trim() !== '') { const requested = parseProviders(raw); - const missing = requested.filter((p) => env[CATALOG[p].apiKeyEnv] === undefined); + const missing = requested.filter((p) => !hasApiKey(env, p)); if (missing.length > 0) { const detail = missing.map((p) => `${p} (${CATALOG[p].apiKeyEnv})`).join(', '); throw new Error(`Missing API key for selected provider(s): ${detail}.`); } return requested; } - const available = PROVIDERS.filter((p) => env[CATALOG[p].apiKeyEnv] !== undefined); + const available = PROVIDERS.filter((p) => hasApiKey(env, p)); if (available.length === 0) { const keys = PROVIDERS.map((p) => CATALOG[p].apiKeyEnv).join(', '); throw new Error(`No provider API key found; set one of ${keys}, or pick providers with EVAL_PROVIDERS.`); diff --git a/packages/evals/test/models.test.ts b/packages/evals/test/models.test.ts index 903537f..195bd4e 100644 --- a/packages/evals/test/models.test.ts +++ b/packages/evals/test/models.test.ts @@ -31,6 +31,23 @@ describe('selectProviders', () => { it('throws when no provider key is set at all', () => { expect(() => selectProviders({})).toThrow(/No provider API key found/); }); + + it('treats a blank key value as absent, like the playground', () => { + expect(selectProviders({ ANTHROPIC_API_KEY: '', ...OPENAI })).toEqual(['openai']); + expect(() => selectProviders({ ANTHROPIC_API_KEY: '', OPENAI_API_KEY: '' })).toThrow( + /No provider API key found/, + ); + }); + + it('rejects an explicitly selected provider whose key is blank', () => { + expect(() => selectProviders({ EVAL_PROVIDERS: 'anthropic', ANTHROPIC_API_KEY: '' })).toThrow( + /ANTHROPIC_API_KEY/, + ); + }); + + it('falls back to key auto-detection when EVAL_PROVIDERS is blank', () => { + expect(selectProviders({ EVAL_PROVIDERS: ' ', ...GOOGLE })).toEqual(['google']); + }); }); describe('benchmarkModels', () => {