feat: add post 'We re-ran it on a real EMR' (OpenEMR head-to-head)#10
Draft
abrichr wants to merge 2 commits into
Draft
feat: add post 'We re-ran it on a real EMR' (OpenEMR head-to-head)#10abrichr wants to merge 2 commits into
abrichr wants to merge 2 commits into
Conversation
Follow-up to the-500th-run: same compiled-vs-agent head-to-head on the official OpenEMR public demo. Both arms 100% (20/20 vs 10/10); compiled 1.8x faster at p50 (39.2s vs 70.4s), 0 vs ~24 model calls/run, $0 vs $0.5522/run at list. Framing: agents don't fail on real apps — repetition cost, determinism, and auditability are the differentiators. Full caveats disclosed (shared mutable demo, N=10, OCR undercount, model pinned, demonstration required), plus an engineering aside on cost guardrails and the ~30% prompt-cache hit rate under screenshot truncation. Numbers from openadapt-flow benchmark/openemr/results.json (2026-07-08). Does not modify the original post. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
Add the run-20 nuance to the caveats section: postconditions flagged drift on the final step, the arm-independent OCR check verified the note saved, and halting on drift is what safe failure looks like on a live shared instance. Matches the BENCHMARK.md disclosure; headline numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Short follow-up post to the-500th-run:
Does not touch the original post. Chart is the latency_cost.png generated by the benchmark run.
Dependencies
Linked files (
benchmark/openemr/BENCHMARK.md,docs/showcase-openemr/FINDINGS.md) land on openadapt-flowmainwith OpenAdaptAI/openadapt-flow#11 (awaiting review). Do not merge/publish this before that one.Test plan
hugo buildpasses locally with the new post🤖 Generated with Claude Code
https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ