feat(auto-router): add JEV classifier alongside LLM classifier

[devin-ai-integration]

TLDR

Adds JEV Classifier alongside the existing LLM Classifier in Auto Router. JEV uses TypeSafe System One to choose a tier, and Auto Router sends the request to that tier's downstream LLM

Problem this solves:

• JEV lacks dashboard configuration and a dedicated connection check
• Paid JEV evaluations need caller context and usage accounting

How it solves it:

• Preserves JEV configuration through creation, editing, and presets
• Probes JEV separately and rejects fallback as connection success
• Shares bounded context, dependency authorization, and TypeSafe accounting

User Flow

Before: an administrator can configure JEV through the API but cannot finish that setup in the dashboard

1. Open https://litellm-domain/ui/?page=models and add an auto-router
2. Configure tier models and choose a classification method
3. JEV is absent from the classification choices, so its settings and connection cannot be checked here

After: the administrator can configure and check JEV in the same dashboard flow

1. Open https://litellm-domain/ui/?page=models and add an auto-router
2. Configure tier models and choose a classification method
3. Select JEV Classifier, set its model and timeout, and click Test Connection
4. The separate JEV result succeeds only when JEV classified the prompt; fallback shows an error while tier probes still run
5. Save and reopen the router with its JEV settings, custom tiers, and context settings preserved, including a preset's per-turn context limit
6. Switching to heuristic and saving removes the obsolete JEV and classifier-context settings
7. Send a Playground message, then open Logs to see the selected tier and separate classifier and completion costs

Before, for encrypted delegated tasks:

1. Send POST https://litellm-domain/v1/responses with a JEV router and an agent_message containing input_text and encrypted_content
2. The response reports x-litellm-complexity-router-cause: jev_classifier and a classifier charge for evaluating the readable envelope

After, for encrypted delegated tasks:

1. Send POST https://litellm-domain/v1/responses with a JEV router and an agent_message containing input_text and encrypted_content
2. Routing uses the configured fallback without a JEV evaluation or classifier charge. The downstream request retains its original task

The TypeSafe credential stays on the server through TYPESAFE_API_KEY and optional trusted TYPESAFE_API_BASE, or through an API-configured router. Dashboard edits preserve stored transport credentials. Saved Test Connection uses the deployment ID to resolve the authorized server configuration

Relevant issues

Builds on #41607 and #41615

Budget admission fix #41879 is merged and included through main

Absorbs backend commit 4913823ccef7cd0d6bc12f1addbc099d3cc267c7 and dashboard commit 060fcbfe1af5640b7f431c1c6fb77fcdda77cfb8, preserving contribution credit

Docs and measured benchmark are merged: BerriAI/litellm-docs#1555. Deploy this feature before announcing JEV dashboard availability

Affected release

Linear ticket

Implementation

Keeps classifier_type: jev, jev_classifier_config, and TypeSafe System One Choice evaluation at /v1/systemone. No new public endpoint or classifier architecture

The dashboard sends complete unsaved configuration and every configured tier to POST /auto_router/test_routing. Saved Test Connection instead supplies saved_model_id and a schema-valid configuration without transport credentials. The server resolves the stored deployment, checks model/team access and availability, and replaces the supplied configuration before classification. Only routing_decision.cause == "jev_classifier" marks JEV connectivity as successful. The diagnostics card includes classifier model, confidence, probabilities, cost, tier, and cause

JEV uses a typesafe/<model> evaluation dependency, separate from completion-model dependencies and their health checks. Default instructions remain available without a license; non-default instructions use the existing customization allowance. JEV and LLM classification share the bounded conversation-context builder

The requester explicitly approved keeping LLM/JEV history parity: three prior user turns within an 8,000-character prior-turn budget, excluding assistant turns by default. JEV sends that context to the configured TypeSafe endpoint. Existing JEV routers inherit those defaults on upgrade; classifier_context_window_size: 0 preserves no-history behavior while still sending the current ask and selected system text. Configuration descriptions, generated API types, dashboard copy and companion documentation disclose this behavior

JEV now uses the shared _encrypted_classifier_task detector before building its request or acquiring a circuit permit. A current encrypted delegated task follows the existing custom-tier, default-model or heuristic fallback. The integration has no native encrypted-task interface, so it neither sends the readable envelope to JEV nor asks JEV to classify a placeholder. This matches the LLM path's fallback for a classifier that cannot handle encrypted tasks; compatible native LLM classifiers still receive their structured task

HTTP status and strict nonnegative token counts are validated before usage dispatch. Valid HTTP-success usage remains billable even when verdict validation fails. The existing TypeSafe handler preserves caller attribution, session and trace linkage, message privacy, and internal-call tagging while stripping parent reservations

Synchronous logging failures no longer discard successful JEV classifications or open the classifier circuit. The warning includes only the exception class. Failed worker submission closes the unsubmitted callback coroutine. HTTP status errors, invalid provider responses and cancellation still propagate

The shared dashboard builder serializes classifier_context_per_turn_chars for creation and editing, including preset values and form overrides. Edit saves preserve untouched values but remove them on incompatible classifier transitions or explicit clearing

Validation

Current feature tip 83ec5d610155b3350f3f3596f2cc2a3d7f764994 includes main fe34fe6047fd6a83afe6984e7bb67e1586e04f92, incorporating merged budget fix #41879 and upstream Heuristic-v2 success-threshold controls. The preceding merge retained both JEV configuration and threshold fields, validation and tests

At this tip, 96 focused JEV and encrypted-task tests pass, including all 43 JEV unit tests. The three new fallback cases first failed without the guard, then passed with it. They assert zero provider calls for the encrypted task, no classifier cost, unchanged original input, and successful JEV classification of a later plaintext ask despite the older encrypted task in history. The request includes a trailing Codex environment reminder so detection must select the actual current task

make check passes Python lint, type-discipline, test-quality, basedpyright budget gates, dashboard lint budgets and generated API/schema synchronization. Dashboard lint has warnings but zero errors. Earlier validation passed 1,414 backend tests and 715 dashboard tests; those broader suites were not repeated locally for this five-line guard. Existing logging fault tests retain provider verdicts after malformed metadata, RuntimeError and TimeoutError

Saved-probe regression coverage includes missing/blocked deployments, model/team authorization, budget denial, non-router IDs and requests without credentials. Full dashboard TypeScript checking remains failing: the preceding merged feature workspace reported 1,356 diagnostics and exact merged main reported 2,084 with the same dependencies. Every normalized feature diagnostic heading also appeared on baseline. No test expectations were changed

The feature includes main's independent replacement for the obsolete login-throttle test. This feature did not change that behavior or test

The saved-probe finding has a code-linked rebuttal. The saved model page uses the saved builder and deployment ID; the cited Add Auto Router form creates new routers. The refreshed browser run below independently confirms the saved request shape

The preceding saved-probe rerun passed 23 dashboard tests and eight authorization/configuration tests. Backend tests separately assert that the stored endpoint and authorization header are used. No code change was needed for that finding

The logging-timeout finding has a source-linked rebuttal. Post-response logging is synchronous; callbacks are queued without awaiting them. The preceding synthetic timing experiment preserved two verdicts with 60ms HTTP delay and 250ms synchronous accounting against a 100ms timeout. Its initial cold-worker run was inconclusive for the second request, which fell back during the HTTP wait before any response

The encrypted-task finding is fixed in 83ec5d6, with a threaded reply. Fresh Bugbot, Greptile and Veria reassessments are requested. CI was still running at this update, with no failed checks; pending reviews are not clearance

Veria's preceding review retained two low-severity findings. Usage validation rejects negative/noninteger token counts before accounting; positive provider usage retains the existing trust model. History preparation matches the LLM defaults explicitly approved by the requester. The requester accepted these documented exceptions; neutral is not a clean bot verdict

Pre-Submission checklist

• I have added meaningful tests
• The focused test files covering the change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is isolated to JEV auto-router integration
• Greptile confidence is at least 4/5 at the current tip before maintainer review

Screenshots / Proof of Fix

Baseline main fe34fe6047fd6a83afe6984e7bb67e1586e04f92 and feature 83ec5d610155b3350f3f3596f2cc2a3d7f764994 use real TypeSafe JEV and Anthropic Haiku APIs. Feature gateways were restarted from this committed checkout. Dashboard 3000 targets gateway 4011; baseline dashboard 3001 targets 4020. The saved feature fixture is 591de6c7-bdd5-4937-ab4b-271c8f4ea8fd. All tiers map to qa-haiku, and provider credentials stay server-side

For the API case, both gateways have an API-configured jev-refresh-8505 router using jev-latest, timeout 12000, ordinary heuristic fallback, no session affinity, context 3/900 and the same tier mappings. Save this synthetic payload as encrypted-task-request.json. $LITELLM_MASTER_KEY authenticates to the local gateway

{
  "model": "jev-refresh-8505",
  "input": [
    {
      "type": "agent_message",
      "author": "/root",
      "recipient": "/root/child",
      "content": [
        {"type": "input_text", "text": "Message Type: NEW_TASK\nPayload:\nSay hello"},
        {"type": "encrypted_content", "encrypted_content": "synthetic-opaque-task"}
      ]
    },
    {"role": "user", "content": "<environment_context>cwd=/repo</environment_context>"}
  ],
  "metadata": {"session_id": "jev-encrypted-before", "user_agent": "codex-tui"},
  "max_output_tokens": 64
}

Before (fe34fe6047fd6a83afe6984e7bb67e1586e04f92)

Dashboard configuration and routing

1. Open http://localhost:3001/models-and-endpoints/ in an isolated browser context
2. Click Auto-Routers, Add Auto Router, Detailed Configuration, then Advanced: Classification Method
3. LLM Classifier is available, but JEV is absent, preventing dashboard JEV setup and connection testing

Encrypted delegated task

1. Send the shared payload to the baseline gateway

   curl -sS http://localhost:4020/v1/responses \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H 'Content-Type: application/json' -H 'User-Agent: codex-tui' \
     --data-binary @encrypted-task-request.json \
     -D before-headers.txt -o before-response.json
   rg -i '^HTTP|^x-litellm-(classifier-cost|complexity-router)' before-headers.txt
   jq '{status,model,output_types:[.output[]?.type]}' before-response.json

2. Observe a paid JEV classification of the envelope

   HTTP/1.1 200 OK
   x-litellm-classifier-cost: 1.8102e-05
   x-litellm-complexity-router-tier: SIMPLE
   x-litellm-complexity-router-cause: jev_classifier
   {"status":"completed","model":"jev-refresh-8505","output_types":["message"]}
   

After (83ec5d610155b3350f3f3596f2cc2a3d7f764994)

Dashboard configuration and routing

1. Open http://localhost:3000/models-and-endpoints/?model=591de6c7-bdd5-4937-ab4b-271c8f4ea8fd and click Edit Auto Router. JEV appears beside LLM, with standard Simple, Medium, Complex and Reasoning tiers
2. Inspect retained settings: jev-latest, timeout 12000, blank instructions, four qa-haiku mappings and context 3/900. Dismiss without saving
3. Click saved Test Connection. The dashboard shows “JEV classification succeeded” and successful downstream connectivity. Both diagnostic requests return HTTP 200, jev_classifier, SIMPLE, confidence 1.0, probabilities and cost $0.000018102. Both carry the exact saved_model_id and no api_key, api_base or masked credentials
4. Open http://localhost:3000/playground/, clear the chat, explicitly select jev-refresh-8505 and send “Say hello.” A real Haiku completion returns a nonempty greeting
5. Open http://localhost:3000/logs/ and expand the newest grouped session. Separate children show TypeSafe classifier cost $0.00001789 and completion cost $0.00009000. Completion details show SIMPLE, JEV, confidence 82%, probabilities and target qa-haiku

Encrypted delegated task

1. Send the identical shared payload to the feature gateway

   curl -sS http://localhost:4011/v1/responses \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H 'Content-Type: application/json' -H 'User-Agent: codex-tui' \
     --data-binary @encrypted-task-request.json \
     -D after-headers.txt -o after-response.json
   rg -i '^HTTP|^x-litellm-(classifier-cost|complexity-router)' after-headers.txt
   jq '{status,model,output_types:[.output[]?.type]}' after-response.json

2. Observe configured fallback and no classifier-cost header

   HTTP/1.1 200 OK
   x-litellm-complexity-router-tier: SIMPLE
   x-litellm-complexity-router-cause: heuristic_scorer
   x-litellm-complexity-router-score: -0.05
   {"status":"completed","model":"jev-refresh-8505","output_types":["message"]}
   

The opaque marker is synthetic, with no private task content. These calls verify classifier bypass and ordinary real-provider completion, not encrypted payload decryption or native Codex behavior. Unit tests independently verify zero outbound JEV requests and original-input preservation

Screenshot review and two-second recording samples showed no plaintext credentials; this was not an exhaustive frame audit. The saved fixture remains unchanged. Stored and environment keys are identical, so this browser run cannot distinguish credential sources

This targeted refresh used the existing unlimited license. One Test Connection action issued two successful requests in the development dashboard, which retains its visible development issue indicator. This run does not establish a clean console or attribution of that warning. It did not repeat fault injection, budget denial, finite customization exhaustion, explicit transport clearing, preset/create/import/export combinations, threshold editing or the historical full matrix. Budget-denial proof remains in merged #41879. Worker-pinned distribution, failover, general durability, actual outbound history and concurrency were not established

Type

New Feature

Caveats (if any)

Severe

• Upgrades enable bounded JEV history sharing; window zero disables history

Medium

• Full dashboard TypeScript checks remain failing
• Concurrent reservation ceilings and crash durability remain unproven
• Failed logging can omit classifier spend for that request
• Encrypted delegated tasks use configured fallback, not JEV classification

Low

• Finite customization exhaustion needs a finite-allowance license
• Browser tests do not exhaust every role and scope
• Malformed usage and explicit clearing have regression-test coverage only
• Veria exceptions accepted; current-tip reviewer reassessments remain pending
• Development Test Connection action produced two successful diagnostic requests
• Development console shows nested-button and empty-image warnings
• Saved nested credentials retain the existing database storage format
• Native Codex and genuine encrypted-payload decryption were not exercised

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world use cases are not possible after this PR

---

Note

Medium Risk
Touches paid external classification, authorization/budget gates, and credential merge on model updates; behavior changes for encrypted tasks and default JEV context on upgrade.

Overview
Adds JEV (TypeSafe System One) as a first-class complexity auto-router classifier alongside the LLM path, with matching dashboard configuration, connection testing, and routing preview.

Backend routing & billing: JEV classification reuses the shared bounded conversation-context builder (same window/budget knobs as LLM), refuses encrypted agent tasks (falls back without calling TypeSafe), and records successful /v1/systemone responses through the existing TypeSafe passthrough logging path so classifier spend inherits caller session/trace metadata. Strategy-router dependencies now include a paid typesafe/<model> evaluation role—authorized for keys/teams and routing tests but excluded from completion health probes.

Proxy & persistence: POST /auto_router/test_routing accepts optional saved_model_id to load server-side router config (with access checks) instead of trusting client-supplied credentials; JEV test authorization derives billable models from strategy_router_dependencies. Patches to complexity_router_config merge stored JEV api_key/api_base when the dashboard sends partial updates. Non-default JEV instructions count toward the existing custom-classifier capability gate.

Dashboard: New JEV classifier UI (model, timeout, circuit breaker, licensed custom instructions), shared context/fallback controls, saved-JEV connection probes (success only when cause === jev_classifier), routing diagnostics, and row labeling for JEV routers.

^{Reviewed by Cursor Bugbot for commit 83ec5d6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/bfc739cc228944f49a2e5017a9b23a66
Open in Devin Desktop: https://app.devin.ai/desktop/session/bfc739cc228944f49a2e5017a9b23a66?variant=devin
Requested by: @moe-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_jev_autorouter_launch_1789767495 (83ec5d6) with main (36b8be7)}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the latest encrypted-task handling avoids contacting JEV and correctly uses each configured fallback mode.

Summary

Adds JEV as an Auto Router classifier with dashboard configuration, saved-router diagnostics, bounded classifier context, dependency authorization, and separate usage accounting.

• The latest change detects encrypted delegated tasks before any JEV call and routes them through the configured fallback.
• Regression coverage verifies fallback-tier, default-model, and heuristic behavior while preserving ordinary plaintext JEV classification.
• Previously reported findings are resolved or withdrawn, and no new actionable issue was identified.

_{Reviews (19) · Last reviewed commit: "fix(auto-router): skip JEV for encrypted..."}

[codecov]

Codecov Report

❌ Patch coverage is 91.57895% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...outer_strategy/complexity_router/jev_classifier.py	82.50%	7 Missing ⚠️
...er_strategy/complexity_router/complexity_router.py	92.85%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[devin-ai-integration]

@veria-ai please review 8e5f43f: JEV dashboard, dependency authorization, bounded context, licensing parity, and TypeSafe accounting, including failed HTTP response handling

[cursor]

Stale Bugbot comment from a previous run.

[veria-ai]

PR overview

This PR adds a JEV classifier alongside the existing LLM classifier in the auto-router’s complexity routing strategy, including support for classifier context payloads.

One issue remains open: JEV classification requests may include recent conversation history by default, exposing prior user turns to the JEV endpoint without explicit opt-in. The impact is conditional on using JEV with multi-turn conversations, and one other issue has already been addressed.

Open issues (1)

• Low: Conversation history disclosed without opt-in — litellm/router_strategy/complexity_router/complexity_router.py:2129

Fixed/addressed: 1 · PR risk: 3/10

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai @veria-ai please review e0b2c51, which validates usage before accounting and prevents stale context restoration when changing classifiers

[mateo-berri]

@greptileai

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@veria-ai Please re-review e0b2c51. Strict usage validation precedes accounting; registry-only TypeSafe pricing preserves billable malformed verdicts. Reassess the remaining finding

[mateo-berri]

bugbot run

[devin-ai-integration]

@veria-ai Please review e0545a7 after the main refresh. Usage is validated before accounting; registered TypeSafe pricing and provider-reported positive usage match existing billing

[devin-ai-integration]

bugbot run

@greptileai @veria-ai please review 93dddbf, incorporating the merged budget fix. Saved-connection behavior is unchanged and targeted regressions pass

[moe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

bugbot run

Please review f9dc57a. Logging failures now preserve JEV verdicts, with regression coverage for metadata failures and timeouts

[devin-ai-integration]

@veria-ai Please reassess f9dc57a. Logging failures preserve valid classifications; provider validation remains enforced. Previously approved history defaults are unchanged

[devin-ai-integration]

@greptileai Please review f9dc57a. Logging failures preserve successful JEV classifications. All 40 JEV tests and make check pass

[moe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

bugbot run

Please review bb46e8b after merging main. JEV configuration and upstream threshold controls both pass regression and real-provider browser tests

[devin-ai-integration]

@greptileai Please reassess bb46e8b after merging main. JEV and threshold controls are preserved; 715 dashboard and 1,414 backend tests pass

[devin-ai-integration]

@veria-ai Please reassess bb46e8b. Main’s threshold controls coexist with JEV configuration. Usage validation and requester-approved history behavior remain unchanged

[moe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai @veria-ai Please reassess 83ec5d6: encrypted delegated tasks use configured fallback without contacting JEV. Regression tests and real-provider dashboard validation pass

[moe-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 83ec5d6. Configure here.}