feat(vault): add separate login and secret namespaces for HashiCorp Vault

[devin-ai-integration]

TLDR

Problem this solves:

• One HCP_VAULT_NAMESPACE drives both Vault login and secret paths
• A customer logs in at root but stores secrets in a team namespace
• No way to express that today, so Vault returns 403 on every secret

How it solves it:

• New HCP_VAULT_LOGIN_NAMESPACE: X-Vault-Namespace header on AppRole and TLS login only
• New HCP_VAULT_SECRET_NAMESPACE: URL path segment on reads, writes, rotates, deletes
• Each falls back to HCP_VAULT_NAMESPACE, so existing deployments are unchanged
• Per-secret secret_manager_settings.namespace still wins over the secret namespace
• Both settings exposed in config, env, the Vault connection test and the Admin UI form
• Docs: docs(vault): document separate login and secret namespaces litellm-docs#1511

User Flow

Before: the proxy admin cannot make Vault login and Vault secret paths use different namespaces, so a team-namespaced secret store never works

1. They set HCP_VAULT_NAMESPACE to the team namespace, say teams/team-a, and start the proxy with AppRole credentials that live in root
2. Startup logs show the AppRole login to Vault failing, because the login request carries X-Vault-Namespace: teams/team-a and the role does not exist there
3. They try the other way round and set HCP_VAULT_NAMESPACE=root (or leave it unset) so login succeeds
4. They send POST http://localhost:4000/key/generate with {"key_alias": "team-a", "models": ["gpt-5.5"]} and get a sk-... key back, but the proxy log shows the Vault write to /v1/secret/data/litellm/team-a returning 403 because the key store is under teams/team-a
5. They send POST http://localhost:4000/v1/chat/completions with that key and get a 401 AuthenticationError: OpenAIException - The api_key client option must be set, because the provider secret is read from /v1/secret/data/OPENAI_API_KEY at root, where it does not exist

After: the same admin logs in at root and keeps every secret under the team namespace

1. They set HCP_VAULT_LOGIN_NAMESPACE=root and HCP_VAULT_SECRET_NAMESPACE=teams/team-a (or type the same two values into the Hashicorp Vault card on the Admin Access page of the Admin UI) and start the proxy with the same AppRole credentials
2. Startup logs show the AppRole login succeeding with X-Vault-Namespace: root
3. They send POST http://localhost:4000/key/generate with {"key_alias": "team-a", "models": ["gpt-5.5"]} and get a sk-... key back, and Vault now holds it at teams/team-a/secret/data/litellm/team-a
4. They send POST http://localhost:4000/v1/chat/completions with that key and get a 200 with the model's reply, because the provider secret is read from /v1/teams/team-a/secret/data/OPENAI_API_KEY

Relevant issues

Supersedes #21759, which was closed as stale before _build_secret_target and per-secret overrides landed

Affected release

Linear ticket

Resolves LIT-1983

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.); documentation waits on docs(vault): document separate login and secret namespaces litellm-docs#1511
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Vault namespaces are an Enterprise feature, and the only Vault available locally is the OSS hashicorp/vault:1.20 image, so namespace enforcement itself was not exercised. What the run below proves is the wire artifact: which namespace the proxy puts on the login request header and which namespace it puts in every secret URL. To make the two arms observably different on OSS Vault, the KV store was laid out the way a namespaced deployment looks from the URL's point of view: on Enterprise a secret in namespace teams/team-a is addressed as /v1/teams/team-a/secret/data/<name>, so the fixture mounts a KV v2 engine at teams/team-a/secret and scopes the AppRole policy to it, while the default secret/ mount stays empty and unreadable by that role

Shared setup, identical for both arms

1. Vault: docker run -d --name vault-lit1983 -p 127.0.0.1:8283:8200 --cap-add=IPC_LOCK hashicorp/vault:1.20 (dev mode), KV v2 mounts at secret/ and teams/team-a/secret/, policy team-a-secrets = path "teams/team-a/secret/*" { capabilities = ["create","read","update","delete","list"] }, AppRole litellm bound to that policy, OPENAI_API_KEY written to teams/team-a/secret/data/OPENAI_API_KEY
2. A tiny local relay on 127.0.0.1:8284 forwards to Vault and appends one line per request (METHOD path -> status headers=...) with X-Vault-Token replaced by <redacted>; it never logs bodies. The log is truncated before each arm
3. Proxy config (lit1983_proxy_config.yaml): one openai/gpt-5.5 deployment with api_key: os.environ/OPENAI_API_KEY, key_management_system: hashicorp_vault, store_virtual_keys: true, prefix_for_stored_virtual_keys: "litellm/", hosted_keys: [OPENAI_API_KEY], master_key: sk-1234
4. Proxy environment for both arms: HCP_VAULT_ADDR=http://127.0.0.1:8284, HCP_VAULT_LOGIN_NAMESPACE=root, HCP_VAULT_SECRET_NAMESPACE=teams/team-a, HCP_VAULT_MOUNT_NAME=secret, HCP_VAULT_APPROLE_ROLE_ID and HCP_VAULT_APPROLE_SECRET_ID from the fixture; OPENAI_API_KEY and HCP_VAULT_NAMESPACE unset
5. Each arm: PYTHONPATH="$TREE:$TREE/enterprise" python litellm/proxy/proxy_cli.py --config /home/ubuntu/lit1983/lit1983_proxy_config.yaml --port <port> --detailed_debug from its own checkout, with litellm.__file__ printed first to prove which tree served the requests

Before (8fc9c46, the merge base)

Split namespaces, login at root and secrets under teams/team-a

1. Confirm the tree

   $ git -C /home/ubuntu/repos/litellm_base rev-parse --short HEAD
   8fc9c46d1a
   $ grep 'litellm.__file__' /home/ubuntu/lit1983/proxy_before4.log
   litellm.__file__ = /home/ubuntu/repos/litellm_base/litellm/__init__.py
   

2. Generate a virtual key (the proxy still answers because the DB write succeeds; the Vault copy fails, see step 4)

   $ curl -s -X POST http://localhost:4183/key/generate -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"key_alias": "lit1983-before4", "models": ["gpt-5.5"]}' | jq '{key_alias, models, key: (.key[:6] + "...")}'
   {
     "key_alias": "lit1983-before4",
     "models": [
       "gpt-5.5"
     ],
     "key": "sk-gly..."
   }
   

3. Call the model with that key: the provider secret cannot be read, so the call fails

   $ curl -s -X POST http://localhost:4183/v1/chat/completions -H "Authorization: Bearer $VIRTUAL_KEY" -H 'Content-Type: application/json' -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Reply with exactly: LIT-1983"}]}' | jq '{model, content: .choices[0].message.content, usage: .usage.total_tokens, error: .error.message}'
   {
     "model": null,
     "content": null,
     "usage": null,
     "error": "litellm.AuthenticationError: AuthenticationError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. Received Model Group=gpt-5.5\nAvailable Model Group Fallbacks=None"
   }
   

4. What the proxy actually sent to Vault: the two new variables are ignored, the login has no namespace header and every secret path is at root, so Vault denies them

   $ cat /home/ubuntu/lit1983/vault_relay.log
   POST /v1/auth/approle/login -> 200 headers={'Content-Type': 'application/json'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/auth/approle/login -> 200 headers={'Content-Type': 'application/json'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/secret/data/litellm/lit1983-before4 -> 403 headers={'x-vault-token': '<redacted>', 'content-type': 'application/json'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   GET /v1/secret/data/OPENAI_API_KEY -> 403 headers={'X-Vault-Token': '<redacted>'}
   

Admin UI, Admin Access page, Hashicorp Vault card

1. Start the proxy from the base tree on port 4000 (ui_proxy_config.yaml, same model list, no Vault) and the dashboard from the base tree with npm run dev on port 3000, sign in as the proxy admin with sk-1234

2. Open http://localhost:3000/admin-panel/, expand the Hashicorp Vault card and click Edit Configuration; the Edit Hashicorp Vault Configuration modal opens

3. The modal shows a single Namespace field; there is nowhere to give login and secrets different namespaces

   Screenshot: Hashicorp Vault modal on the merge base, one Namespace field

After (efd2b8a, the PR tip)

Split namespaces, login at root and secrets under teams/team-a

1. Confirm the tree

   $ git -C /home/ubuntu/repos/litellm rev-parse --short HEAD
   efd2b8a3cd
   $ grep 'litellm.__file__' /home/ubuntu/lit1983/proxy_after4.log
   litellm.__file__ = /home/ubuntu/repos/litellm/litellm/__init__.py
   

2. Generate a virtual key

   $ curl -s -X POST http://localhost:4184/key/generate -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"key_alias": "lit1983-after4", "models": ["gpt-5.5"]}' | jq '{key_alias, models, key: (.key[:6] + "...")}'
   {
     "key_alias": "lit1983-after4",
     "models": [
       "gpt-5.5"
     ],
     "key": "sk--Ze..."
   }
   

3. Call the model with that key: real OpenAI call, real tokens billed

   $ curl -s -X POST http://localhost:4184/v1/chat/completions -H "Authorization: Bearer $VIRTUAL_KEY" -H 'Content-Type: application/json' -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Reply with exactly: LIT-1983"}]}' | jq '{model, content: .choices[0].message.content, usage: .usage.total_tokens, error: .error.message}'
   {
     "model": "gpt-5.5",
     "content": "LIT-1983",
     "usage": 34,
     "error": null
   }
   

4. What the proxy actually sent to Vault: login carries X-Vault-Namespace: root, the provider secret read and the virtual key write both address teams/team-a in the URL, and no secret request carries a namespace header

   $ cat /home/ubuntu/lit1983/vault_relay.log
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'root', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'root', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/teams/team-a/secret/data/litellm/lit1983-after4 -> 200 headers={'x-vault-token': '<redacted>', 'content-type': 'application/json'}
   

Admin UI, Admin Access page, Hashicorp Vault card

1. Start the proxy from the PR tree on port 4000 (ui_proxy_config.yaml, same model list, no Vault) and the dashboard from the PR tree with npm run dev on port 3000, sign in as the proxy admin with sk-1234

2. Open http://localhost:3000/admin-panel/, expand the Hashicorp Vault card and click Edit Configuration; the Edit Hashicorp Vault Configuration modal opens

3. The modal now shows Login Namespace and Secret Namespace next to the existing Namespace field

   Screenshot: Hashicorp Vault modal on the PR tip, Login Namespace and Secret Namespace fields empty

4. Fill Namespace root, Login Namespace root, Secret Namespace teams/team-a; the vitest in EditHashicorpVaultModal.test.tsx asserts these same three fields reach the save payload as vault_namespace, vault_login_namespace and vault_secret_namespace

   Screenshot: Hashicorp Vault modal on the PR tip with the three namespace fields filled

Backward compatibility, legacy config at base and tip

Same fixture, but the proxy environment carries only the legacy variable, HCP_VAULT_NAMESPACE=teams/team-a, with HCP_VAULT_LOGIN_NAMESPACE and HCP_VAULT_SECRET_NAMESPACE unset. This is the path every existing deployment is on. Both trees log in with X-Vault-Namespace: teams/team-a, read and write under teams/team-a in the URL, and the model call returns LIT-1983 with 34 tokens billed

1. Merge base, 8fc9c46, served from /home/ubuntu/repos/litellm_base, key alias lit1983-legacybase

   $ cat /home/ubuntu/lit1983/vault_relay.log
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'teams/team-a', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'teams/team-a', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/teams/team-a/secret/data/litellm/lit1983-legacybase -> 200 headers={'x-vault-token': '<redacted>', 'content-type': 'application/json'}
   

2. PR tip, efd2b8a, served from /home/ubuntu/repos/litellm, key alias lit1983-legacytip4

   $ cat /home/ubuntu/lit1983/vault_relay.log
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'teams/team-a', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/auth/approle/login -> 200 headers={'X-Vault-Namespace': 'teams/team-a', 'Content-Type': 'application/json'}
   GET /v1/teams/team-a/secret/data/OPENAI_API_KEY -> 200 headers={'X-Vault-Token': '<redacted>'}
   POST /v1/teams/team-a/secret/data/litellm/lit1983-legacytip4 -> 200 headers={'x-vault-token': '<redacted>', 'content-type': 'application/json'}
   

Type

🆕 New Feature

Caveats (if any)

Medium

• Vault Enterprise namespace enforcement was not exercised; the proof shows the wire artifact on OSS Vault
• Secret reads now honour per-secret secret_manager_settings (namespace, mount, path_prefix, data) the same way writes already did; before, reads ignored them and always used the global defaults
• The in-process secret cache is now keyed by the resolved URL and stores the whole KV response body; the selected data field is picked at read time. The same secret name in two namespaces no longer collides, two fields of one secret share one entry, and rotate and delete evict that URL entry so no field of the mutated path can be served stale
• The required documentation check (and code-quality) fails with Environment variables read under ./litellm but mentioned nowhere in the docs: ['HCP_VAULT_LOGIN_NAMESPACE', 'HCP_VAULT_SECRET_NAMESPACE'] until docs(vault): document separate login and secret namespaces litellm-docs#1511 merges, because both workflows check out litellm-docs at its default branch; the same test passes locally against that docs branch

Low

• vault_login_namespace and vault_secret_namespace are stored in the existing hashicorp_secret_manager config override row, no new DB column
• X-Vault-Namespace was never sent on secret reads or writes on main (only the URL segment), so this PR keeps that single mechanism rather than introducing a second one

Notes for reviewers

Root cause in one sentence: HCP_VAULT_NAMESPACE was the only namespace input and it fed both the login header and the secret URL, so a role that lives in one namespace could never manage secrets in another

Security verdict: this is ordinary feature work, not a vulnerability fix. It only changes which namespace already-held Vault credentials authenticate against and read from; it grants nobody access they did not already have, and Vault still enforces the AppRole policy on every request. No token or secret value is logged anywhere in the change or the proof

Taxonomy audit of the diff (items with a concrete changed line, fixed or judged not applicable). A1/A2 fixed: the cache is keyed by the resolved URL and holds the full response body, so a different namespace cannot collide, a different data selector reads its own field from the cached body, and rotate or delete evicts every field of the path at once; one test reads two fields of one path and asserts each comes back, another caches a field, deletes the secret and asserts the next read refetches. W1 fixed: the connection test used to mutate the mapping returned by _get_request_headers; it now builds a separate MappingProxyType and a test asserts the original mapping is untouched. X1-X5 checked: the fallbacks use is not None so an explicitly empty login or secret namespace is honoured as "no namespace" instead of silently falling back to HCP_VAULT_NAMESPACE; _get_login_headers treats empty as "omit the header", which is how Vault addresses root. C1-C7 checked: no default flips, precedence for existing deployments is unchanged because both new values fall back to the legacy one. F3 covered: sync and async reads, write, rotate, delete and both login methods all route through the same two properties; the connection test endpoint was the sibling surface and is included. T1 covered: every namespace test in test_hashicorp_secret_manager.py fails on main (the login header and URL assertions differ), and the tests assert the artifacts the HTTP client received via respx, not the helper's return value. H3/H4: no new source comments, X | None, no bare dict or Any in new code (_get_secret_value_from_json_response takes Mapping[str, object] | None and narrows each nested level through _as_json_object), schema.d.ts and the lazy OpenAPI snapshot regenerated via npm run gen:api and python -m litellm.proxy._lazy_openapi_snapshot. M2 not applicable: no new secret is logged or returned. R not applicable: no migration, the config override table already stores the whole Vault block as JSON

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• Live base vs tip A/B on efd2b8a against 8fc9c46: dependents of the changed symbols are the proxy boot path, the secret read path behind os.environ/ model params, the virtual key write, rotate and delete hooks, the Vault connection test endpoint and the Admin UI form; reads, the key write and both login shapes (split namespaces, legacy single namespace) were driven live at base and tip with the same fixture and real provider calls, the Admin UI modal was opened on both trees, rotate and delete are covered by the respx tests that assert the URL each one hits

Link to Devin session: https://app.devin.ai/sessions/f6aabc2074cc4c5d9ccefa90a49d3636
Open in Devin Desktop: https://app.devin.ai/desktop/session/f6aabc2074cc4c5d9ccefa90a49d3636?variant=devin
Requested by: @yassin-berriai

Link to Devin session: https://app.devin.ai/sessions/798e6e6460fc40deb35c1219944662cf
Open in Devin Desktop: https://app.devin.ai/desktop/session/798e6e6460fc40deb35c1219944662cf?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mateo-berri]

bugbot run

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_vault_login_secret_namespace (efd2b8a) with main (8fc9c46)}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with the previously reported cache issues fully addressed and no new actionable defects identified

Summary

This PR separates HashiCorp Vault login and secret namespaces while preserving the legacy namespace fallback

• Adds independent environment variables and configuration fields for login and secret namespaces
• Applies the login namespace to AppRole, TLS, and connection-test requests
• Applies the secret namespace and per-secret overrides consistently across reads, writes, rotations, and deletions
• Keys cached Vault responses by resolved URL and selects individual data fields at read time
• Exposes the new settings through the Admin UI and generated API schemas
• Adds coverage for namespace precedence, authentication methods, cache isolation, invalidation, and UI payloads

_{Reviews (5) · Last reviewed commit: "refactor(vault): type the KV read body w..."}

[cursor]

Stale Bugbot comment from a previous run.

[codecov]

Codecov Report

❌ Patch coverage is 90.74074% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...itellm/secret_managers/hashicorp_secret_manager.py	89.79%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8691a1e. Configure here.}

[devin-ai-integration]

The documentation and code-quality failures only need BerriAI/litellm-docs#1511 merged first, since CI reads litellm-docs main. No code change needed here

[devin-ai-integration]

@veria-ai please review 8691a1e: two new Vault namespace settings for login header and secret URL, legacy fallback, URL keyed secret cache and its invalidation, connection test headers, Admin UI fields

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[devin-ai-integration]

@veria-ai please review bc636bc: main merged in, same Vault login and secret namespace split, URL keyed cache, connection test headers, Admin UI fields

[mateo-berri]

bugbot run

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@veria-ai please review efd2b8a: main merged in, and the KV read body walk is now typed as Mapping[str, object] with isinstance narrowing