docs: expand JEV Auto Router guidance and publish measured classifier benchmark - #1555
devin-ai-integration[bot] wants to merge 11 commits into
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
bugbot run |
|
@greptileai @veria-ai Please review the JEV setup guidance and benchmark claims against the archived measurements, methodology, limitations, and linked implementation |
|
@veria-ai Please review d5c444d for setup accuracy, benchmark evidence, licensing parity, and credential handling in the JEV documentation and blog |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
|
@greptileai @veria-ai Please review 13f7452: the headline now uses the measured 5.43x median classification speed ratio, with methodology retained below |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 13f7452. Configure here.
|
@greptileai @veria-ai Please review current head 13f7452 for setup accuracy and benchmark claims. Earlier requests produced no review or check result |
Summary
Document the existing
classifier_type: jev/jev_classifier_configpath through TypeSafe System One Choice, including dashboard setup, bounded prior-turn context, timeout recovery, dependency authorization and separate classifier accounting. Behavior was checked against the integrated source in BerriAI/litellm#41886 and the benchmark's budget-first rehearsal treeAdd an engineering post comparing JEV with Haiku on 80 authored synthetic cases, three paired repeats each. The title is “JEV Classifier: 5.43x as Fast as Haiku, 96% Lower Cost”. The body defines the speed ratio using measured median classifier latency, 688.40 ms / 126.81 ms. The post separates authored-label accuracy from agreement and scopes the 96.118% saving to registry-priced classifier cost, with p95, per-tier results, clustered uncertainty and limitations
Publish the unchanged sanitized evidence archive with SHA256 verification and reproduction instructions, including unsuccessful setup attempts. Expand the existing setup page and crosslink the overview, benchmarks, evaluation, proxy reference and TypeSafe page
Validation
npm run lint:writing,npm run lint:docs, the structural checker on the new blog directory,npm run build, andgit diff --checkpassed. The docs check used the existing LiteLLM Python environment for PyYAML. The build emitted nonfatal warnings, including optional canvas resolution and HTML diagnostics on other pagesThe archive matches SHA256
c2653861ab4cf591e902d4d62d55078187e4c66ad5845b5cf5b023d5f427fd7f, its internal manifest verified, and a credential-pattern scan found no matchesCaveats
The corpus has no independent labels, blind adjudication or downstream answer scoring. The eight-case boundary subset's accuracy-difference interval includes zero. The concurrency-one run does not establish production throughput, total application savings or invoice amounts
Real-provider browser validation and refreshed screenshots are published in BerriAI/litellm#41886. Bugbot reports no new issues at current docs head
13f7452a; writing-style and structure checks also pass. Greptile and Veria have not returned reviews or check results despite requests; they were requested again on this head using the authorized human identity. Their absence is not clearance. Publish after #41879 and #41886, once the JEV history-default decision and required reviews are settledNo release number is asserted. The blog theme has a hiring link on its list page, but no automatic post-footer CTA was found, so this PR adds no inline CTA or theme changes
Link to Devin session: https://app.devin.ai/sessions/9a726fd27a444bb59f899353ea912eba
Open in Devin Desktop: https://app.devin.ai/desktop/session/9a726fd27a444bb59f899353ea912eba?variant=devin
Requested by: @moe-berri
Note
Low Risk
Documentation and published benchmark artifacts only; no application or proxy runtime code changes in this diff.
Overview
Adds a new engineering post comparing TypeSafe JEV vs Claude Haiku as Auto Router classifiers on 80 authored cases (5.43× median latency, ~96% lower registry-priced classifier cost, methodology, uncertainty, limitations) and links a SHA256-verified evidence archive for reproduction.
Expands Auto Router docs so
classifier_type: jevis documented end-to-end: dashboard/YAML setup,classifier_context_budget_charsshared with the LLM classifier, circuit breaker and fallback, Test Routing spend, and classifier accounting (separate spend rows vsclassifier_costmetadata,x-litellm-classifier-cost,cause: jev_classifier). The benchmarks index, overview results table, evaluate page, TypeSafe pass-through, and proxy reference are cross-linked to the new post.Reviewed by Cursor Bugbot for commit 13f7452. Bugbot is set up for automated code reviews on this repo. Configure here.
Link to Devin session: https://app.devin.ai/sessions/bfc739cc228944f49a2e5017a9b23a66
Open in Devin Desktop: https://app.devin.ai/desktop/session/bfc739cc228944f49a2e5017a9b23a66?variant=devin