fix(proxy): enforce virtual key budgets for JEV test routing by devin-ai-integration[bot] · Pull Request #41879 · BerriAI/litellm

devin-ai-integration · 2026-09-18T21:06:41Z

TLDR

Problem this solves:

JEV Test Routing accepted zero and exhausted virtual-key budgets
Exhausted throttle-enabled keys could still call JEV

How it solves it:

Apply the existing key-budget check before JEV evaluation
Reject JEV previews when that check sets budget_throttle_pct
Preserve free heuristic tests and classifier model-access checks

User Flow

Before: a proxy admin's restricted virtual key can run paid JEV tests after its budget is exhausted

Create an admin-owned virtual key with max_budget=0, rpm_limit=100 and metadata.throttle_on_budget_exceeded=true
Send POST http://127.0.0.1:4011/auto_router/test_routing with "classifier_type":"jev"
Receive HTTP 200 with "cause":"jev_classifier"
Another holder of that exhausted key can repeat these paid evaluations

After: the same key receives the existing budget error before JEV evaluation

Create an admin-owned virtual key with max_budget=0, rpm_limit=100 and metadata.throttle_on_budget_exceeded=true
Send POST http://127.0.0.1:4011/auto_router/test_routing with "classifier_type":"jev"
Receive HTTP 400 with "type":"budget_exceeded"
Another holder of that exhausted key can no longer run these evaluations

Relevant issues

Builds on #41607 and #41615. Merge before #41886

Affected release

Linear ticket

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Tip 3b0d32ec6f14979687d4bb76f51db5f9427a131a: all 176 endpoint tests and make check pass, including Python lint/type checks, test-quality checks and generated API-type synchronization. On preceding tip 503ab0b373, the new zero/exhausted throttle regressions failed with DID NOT RAISE, while funded/unlimited controls passed. Denied cases assert no provider invocation

At this exact tip, CI reports 88 passed, zero failed, zero pending and one skipped check. Greptile reports 5/5, Bugbot reports no new issues, and Veria reports no open security issues with risk 0/10. The final review-thread sweep found no unresolved threads. Current main independently replaced the obsolete login-throttle test; this PR does not modify it

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review)

Screenshots / Proof of Fix

Refreshed live proof on September 19 compares merge base 4487a9915b30b7188f254630bb0f0294800d3072 with fixed tip 3b0d32ec6f14979687d4bb76f51db5f9427a131a. Both ran against an isolated local PostgreSQL database and the real TypeSafe System One API. TypeSafe credentials came from the QA vault. The master key created local admin users and virtual keys; every routing request used a distinct virtual key

The proxy's budget_exceeded_throttle_percentage was 0.1. Each key had rpm_limit=100 and metadata.throttle_on_budget_exceeded=true. The four budget/spend pairs were (0,0), (1,2), (1,0) and (null,0)

Reproduce the calls against each revision:

URL=http://127.0.0.1:4011
curl -sS "$URL/user/new" -H "Authorization: Bearer $QA_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"user_id":"jev-throttle-admin","user_role":"proxy_admin","auto_create_key":false}'

KEY="$(curl -sS "$URL/key/generate" -H "Authorization: Bearer $QA_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"user_id":"jev-throttle-admin","models":[],"max_budget":0,"spend":0,"rpm_limit":100,"metadata":{"throttle_on_budget_exceeded":true}}' \
  | jq -er '.key')"

curl -sS -w '\nHTTP %{http_code}\n' "$URL/auto_router/test_routing" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"prompt":"What is 2+2?","complexity_router_config":{"classifier_type":"jev","jev_classifier_config":{"model":"jev-latest","timeout_ms":10000},"tiers":{"SIMPLE":["qa-nano"],"MEDIUM":["qa-nano"],"COMPLEX":["qa-nano"],"REASONING":["qa-nano"]}}}'

qa-nano was configured as openai/gpt-5.4-nano. Test Routing only classified; it did not call that completion deployment. Repeat key creation with the remaining budget/spend pairs above

Throttle-enabled key	Before `4487a9915b`	After `3b0d32ec6f`
Zero budget	HTTP 200, `jev_classifier`	HTTP 400, `budget_exceeded`
Exhausted positive budget	HTTP 200, `jev_classifier`	HTTP 400, `budget_exceeded`
Remaining budget	HTTP 200, `jev_classifier`	HTTP 200, `jev_classifier`
Unlimited budget	HTTP 200, `jev_classifier`	HTTP 200, `jev_classifier`

The after-run error for the throttle case is:

{"error":{"message":"Budget has been exceeded! JEV Test Routing requires available budget.","type":"budget_exceeded","param":null,"code":"400"}}

The UI/gateway rehearsal for #41886 separately verified ordinary zero budgets, funded-key spend settlement, and a tiny budget that passed once then rejected the next request. This PR adds admission only; classifier accounting is in #41886

Type

Bug Fix

Caveats (if any)

Severe

JEV Test Routing now rejects zero or exhausted key budgets, including throttle-enabled keys

Low

Admission uses existing spend; this adds no spend accounting
Live proof used admin-owned virtual keys, with no non-admin claim
Concurrent reservation ceilings and parent-owned budget combinations were not established by these calls

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Changes proxy authorization on a management endpoint that gates paid JEV evaluations; scope is narrow but affects budget and throttle behavior for virtual keys.

Overview
JEV auto-router test routing (POST /auto_router/test_routing) no longer skips virtual-key budget checks when the config uses a JEV classifier. Previously, _authorize_models_this_test_can_call returned early when there was no billable LLM/embedding model in the test path, so zero or exhausted keys could still invoke the paid JEV provider.

The guard now continues for classifier_type == "jev", runs the existing _virtual_key_max_budget_check, and hard-blocks keys that would normally be throttled after budget exhaustion (budget_throttle_pct set) with a dedicated budget_exceeded error. Heuristic dry runs stay free (unchanged behavior, now covered by parametrized tests).

New unit tests assert denial before JevClassifierClient.evaluate for zero/exhausted budgets and for throttle-enabled keys, while allowing runs when budget remains or is unlimited.

^{Reviewed by Cursor Bugbot for commit 3b0d32e. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/a5c5a748d13446f6aac6f5841dfc283b
Open in Devin Desktop: https://app.devin.ai/desktop/session/a5c5a748d13446f6aac6f5841dfc283b?variant=devin
Requested by: @moe-berri

Link to Devin session: https://app.devin.ai/sessions/bfc739cc228944f49a2e5017a9b23a66
Open in Devin Desktop: https://app.devin.ai/desktop/session/bfc739cc228944f49a2e5017a9b23a66?variant=devin

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-09-18T21:06:44Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

codspeed · 2026-09-18T21:09:03Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_jev_test_budget_1789764884 (3b0d32e) with main (2886b8e)}

codecov · 2026-09-18T21:09:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-09-18T21:09:52Z

Confidence Score: 5/5

The PR appears safe to merge because the new guard narrowly blocks exhausted throttled keys before paid JEV evaluation while preserving valid and free routing paths

Summary

This PR closes the remaining JEV test-routing budget bypass for virtual keys using budget throttling

Rejects JEV evaluation when an exhausted key has been marked for throttling
Preserves successful JEV routing for keys with available or unlimited budget
Adds regression coverage proving denied requests never invoke the JEV provider
Keeps zero-cost heuristic previews available to exhausted keys

_{Reviews (4) · Last reviewed commit: "fix(proxy): reject throttled exhausted b..."}

mateo-berri · 2026-09-18T21:21:50Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

devin-ai-integration · 2026-09-18T21:54:16Z

The proxy-infra failure reproduces on unchanged base 6fa34a299b with the same assertion; this PR does not touch login settings

devin-ai-integration · 2026-09-18T21:54:46Z

@veria-ai please review 0ac0362: JEV Test Routing now checks existing virtual-key budgets before evaluation, with provider-invocation regression coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-18T22:04:13Z

bugbot run

devin-ai-integration · 2026-09-18T22:04:13Z

@veria-ai please review b9e5bb3: the budget admission fix is unchanged; its regression fixture now permits the companion feature's evaluation dependency

devin-ai-integration · 2026-09-18T22:04:21Z

@greptileai review the updated fixture and admission scope. Successful-preview spend accounting lives in companion #41886, which will ship alongside this fix

cursor

Stale Bugbot comment from a previous run.

…et_1789764884

mateo-berri · 2026-09-19T17:41:18Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

devin-ai-integration · 2026-09-19T17:50:22Z

@greptileai @veria-ai Please review 503ab0b after the main refresh. Budget admission and invocation-count regression tests remain unchanged

veria-ai · 2026-09-19T18:02:45Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-19T18:12:38Z

bugbot run

devin-ai-integration · 2026-09-19T18:12:49Z

@greptileai @veria-ai Please review 3b0d32e: JEV previews hard-block exhausted throttled keys. The regression failed before, all 176 endpoint tests and make check pass

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3b0d32e. Configure here.}

Backport of upstream BerriAI#41879 (merge 1e161f5). A Jev routing test calls the paid TypeSafe API even when no tier model is invoked, so the key-budget check now runs for classifier_type 'jev' and refuses exhausted throttle-enabled keys. Tests adapted: preview_auto_router_routing takes no http_request on 1.101. Drop this commit when rebasing onto a release that contains BerriAI#41879. (cherry picked from commit 1e161f5) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Backport of upstream BerriAI#41739 (open, unmerged; commit 36601dd), fixing BerriAI#41713. JevClassifierConfig now requires api_key or TYPESAFE_API_KEY, so /auto_router/validate_complexity_router_config reports the missing key and /auto_router/test_routing refuses the config instead of answering a bare 500. Over HTTP that refusal is a 422 from request-body validation (the proxy maps validation errors to 400 only under /management/v1); the ComplexityRouter construction moved inside the try only matters for configs that bypass body validation. Adapted to the 1.101 base: - The team-member permission tests from the PR are not carried (auto_router_permissions.py does not exist on 1.101). - preview_auto_router_routing takes no http_request on 1.101. - The BerriAI#41879 budget tests now set TYPESAFE_API_KEY, which this validator requires; the two upstream PRs were written independently. Drop this commit when rebasing onto a release that contains BerriAI#41713's fix. (cherry picked from commit 36601dd) Co-Authored-By: wofiporia <1029687661@qq.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Prerequisite for #41615 and #41879 on stable/1.102.x. Cherry-picked from 109ca70 (main).

Backport of #41879 to stable/1.102.x. Cherry-picked from 1e161f5 (main).

fix(proxy): enforce virtual key budgets for JEV test routing

0ac0362

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team September 18, 2026 21:06

devin-ai-integration Bot assigned moe-berri Sep 18, 2026

greptile-apps Bot reviewed Sep 18, 2026

View reviewed changes

Comment thread litellm/proxy/management_endpoints/auto_router_endpoints.py

cursor Bot reviewed Sep 18, 2026

View reviewed changes

devin-ai-integration Bot mentioned this pull request Sep 18, 2026

feat(auto-router): add JEV classifier alongside LLM classifier #41886

Merged

6 tasks

test(proxy): allow JEV dependency in budget fixtures

b9e5bb3

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Sep 18, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into litellm_jev_test_budg…

503ab0b

…et_1789764884

cursor Bot reviewed Sep 19, 2026

View reviewed changes

veria-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread litellm/proxy/management_endpoints/auto_router_endpoints.py

fix(proxy): reject throttled exhausted budgets in JEV previews

3b0d32e

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Sep 19, 2026

View reviewed changes

n24q02m mentioned this pull request Sep 20, 2026

Jev classifier: confidence-gated route-down + fail-expensive option #42139

Open

moe-berri approved these changes Sep 20, 2026

View reviewed changes

moe-berri merged commit 1e161f5 into main Sep 20, 2026
92 checks passed

moe-berri deleted the litellm_jev_test_budget_1789764884 branch September 20, 2026 17:10

devin-ai-integration Bot pushed a commit that referenced this pull request Sep 22, 2026

feat(auto-router): allow opted-in team members to manage their routers

1a3cec8

Prerequisite for #41615 and #41879 on stable/1.102.x. Cherry-picked from 109ca70 (main).

devin-ai-integration Bot pushed a commit that referenced this pull request Sep 22, 2026

fix(proxy): enforce virtual key budgets for JEV test routing

92a2a2f

Backport of #41879 to stable/1.102.x. Cherry-picked from 1e161f5 (main).

devin-ai-integration Bot mentioned this pull request Sep 22, 2026

feat(typesafe): backport #41607 to stable/1.102.x for v1.102.1 #42595

Open

12 tasks

Uh oh!

Conversation

devin-ai-integration Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Caveats (if any)

Severe

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Devin AI Engineer

Uh oh!

codspeed Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Sep 18, 2026

Codecov Report

Uh oh!

greptile-apps Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

mateo-berri commented Sep 18, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot commented Sep 18, 2026

Uh oh!

devin-ai-integration Bot commented Sep 18, 2026

Uh oh!

mateo-berri commented Sep 18, 2026

Uh oh!

devin-ai-integration Bot commented Sep 18, 2026

Uh oh!

devin-ai-integration Bot commented Sep 18, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 19, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot commented Sep 19, 2026

Uh oh!

Uh oh!

veria-ai Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

mateo-berri commented Sep 19, 2026

Uh oh!

devin-ai-integration Bot commented Sep 19, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

devin-ai-integration Bot commented Sep 18, 2026 •

edited

Loading

devin-ai-integration Bot commented Sep 18, 2026 •

edited

Loading

codspeed Bot commented Sep 18, 2026 •

edited

Loading

greptile-apps Bot commented Sep 18, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

veria-ai Bot commented Sep 19, 2026 •

edited

Loading