Skip to content

docs: expand JEV Auto Router guidance and publish measured classifier benchmark - #1555

Open
devin-ai-integration[bot] wants to merge 11 commits into
mainfrom
litellm_jev_autorouter_blog_1789770966
Open

devin-ai-integration[bot] wants to merge 11 commits into
mainfrom
litellm_jev_autorouter_blog_1789770966

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Document the existing classifier_type: jev / jev_classifier_config path through TypeSafe System One Choice, including dashboard setup, bounded prior-turn context, timeout recovery, dependency authorization and separate classifier accounting. Behavior was checked against the integrated source in BerriAI/litellm#41886 and the benchmark's budget-first rehearsal tree

Add an engineering post comparing JEV with Haiku on 80 authored synthetic cases, three paired repeats each. The title is “JEV Classifier: 5.43x as Fast as Haiku, 96% Lower Cost”. The body defines the speed ratio using measured median classifier latency, 688.40 ms / 126.81 ms. The post separates authored-label accuracy from agreement and scopes the 96.118% saving to registry-priced classifier cost, with p95, per-tier results, clustered uncertainty and limitations

Publish the unchanged sanitized evidence archive with SHA256 verification and reproduction instructions, including unsuccessful setup attempts. Expand the existing setup page and crosslink the overview, benchmarks, evaluation, proxy reference and TypeSafe page

Validation

npm run lint:writing, npm run lint:docs, the structural checker on the new blog directory, npm run build, and git diff --check passed. The docs check used the existing LiteLLM Python environment for PyYAML. The build emitted nonfatal warnings, including optional canvas resolution and HTML diagnostics on other pages

The archive matches SHA256 c2653861ab4cf591e902d4d62d55078187e4c66ad5845b5cf5b023d5f427fd7f, its internal manifest verified, and a credential-pattern scan found no matches

Caveats

The corpus has no independent labels, blind adjudication or downstream answer scoring. The eight-case boundary subset's accuracy-difference interval includes zero. The concurrency-one run does not establish production throughput, total application savings or invoice amounts

Real-provider browser validation and refreshed screenshots are published in BerriAI/litellm#41886. Bugbot reports no new issues at current docs head 13f7452a; writing-style and structure checks also pass. Greptile and Veria have not returned reviews or check results despite requests; they were requested again on this head using the authorized human identity. Their absence is not clearance. Publish after #41879 and #41886, once the JEV history-default decision and required reviews are settled

No release number is asserted. The blog theme has a hiring link on its list page, but no automatic post-footer CTA was found, so this PR adds no inline CTA or theme changes

Link to Devin session: https://app.devin.ai/sessions/9a726fd27a444bb59f899353ea912eba
Open in Devin Desktop: https://app.devin.ai/desktop/session/9a726fd27a444bb59f899353ea912eba?variant=devin
Requested by: @moe-berri


Note

Low Risk
Documentation and published benchmark artifacts only; no application or proxy runtime code changes in this diff.

Overview
Adds a new engineering post comparing TypeSafe JEV vs Claude Haiku as Auto Router classifiers on 80 authored cases (5.43× median latency, ~96% lower registry-priced classifier cost, methodology, uncertainty, limitations) and links a SHA256-verified evidence archive for reproduction.

Expands Auto Router docs so classifier_type: jev is documented end-to-end: dashboard/YAML setup, classifier_context_budget_chars shared with the LLM classifier, circuit breaker and fallback, Test Routing spend, and classifier accounting (separate spend rows vs classifier_cost metadata, x-litellm-classifier-cost, cause: jev_classifier). The benchmarks index, overview results table, evaluate page, TypeSafe pass-through, and proxy reference are cross-linked to the new post.

Reviewed by Cursor Bugbot for commit 13f7452. Bugbot is set up for automated code reviews on this repo. Configure here.

Link to Devin session: https://app.devin.ai/sessions/bfc739cc228944f49a2e5017a9b23a66
Open in Devin Desktop: https://app.devin.ai/desktop/session/bfc739cc228944f49a2e5017a9b23a66?variant=devin

moe-berri and others added 8 commits September 18, 2026 22:46
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

devin-ai-integration Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@vercel

vercel Bot commented Sep 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
litellm Ready Ready Preview Sep 18, 2026 10:51pm UTC

Request Review

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai @veria-ai Please review the JEV setup guidance and benchmark claims against the archived measurements, methodology, limitations, and linked implementation

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

@veria-ai Please review d5c444d for setup accuracy, benchmark evidence, licensing parity, and credential handling in the JEV documentation and blog

moe-berri and others added 3 commits September 19, 2026 01:21
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai @veria-ai Please review 13f7452: the headline now uses the measured 5.43x median classification speed ratio, with methodology retained below

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 13f7452. Configure here.

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai @veria-ai Please review current head 13f7452 for setup accuracy and benchmark claims. Earlier requests produced no review or check result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants