Skip to content

Commit 4d0b1a1

Browse files
authored
docs: document LMCache Prometheus metrics harmless error for vLLM 0.12.0 (#5026)
Signed-off-by: Keiven Chang <[email protected]> Co-authored-by: Keiven Chang <[email protected]>
1 parent bf9c93c commit 4d0b1a1

File tree

2 files changed

+45
-0
lines changed

2 files changed

+45
-0
lines changed

docs/backends/vllm/LMCache_Integration.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,45 @@ When LMCache is enabled with `--connector lmcache` and `DYN_SYSTEM_PORT` is set,
160160

161161
For detailed information on LMCache metrics, including the complete list of available metrics and how to access them, see the **[LMCache Metrics section](prometheus.md#lmcache-metrics)** in the vLLM Prometheus Metrics Guide.
162162

163+
### Troubleshooting
164+
165+
#### LMCache log: `PrometheusLogger instance already created with different metadata`
166+
167+
You may see an error like:
168+
169+
```text
170+
LMCache ERROR: PrometheusLogger instance already created with different metadata. This should not happen except in test
171+
```
172+
173+
**Version note**: We reproduced this behavior with **vLLM v0.12.0**. We have not reproduced it with **vLLM v0.11.0**, so it may be specific to (or introduced in) v0.12.0.
174+
175+
This is emitted by LMCache when the LMCache connector is initialized more than once in the same process (for example, once for a `WORKER` role and later for a `SCHEDULER` role). LMCache uses a process-global singleton for its Prometheus logger, so the second initialization can log this warning if its metadata differs.
176+
177+
- **Impact**: This is a log-only error; in our testing it does not prevent vLLM/Dynamo from serving requests. If you care about LMCache metric labels, be aware the logger singleton uses the first-seen metadata.
178+
- **Repro without Dynamo** (vLLM v0.12.0):
179+
180+
```bash
181+
vllm serve Qwen/Qwen3-0.6B \
182+
--host 127.0.0.1 --port 18000 \
183+
--gpu-memory-utilization 0.24 \
184+
--enforce-eager \
185+
--no-enable-prefix-caching \
186+
--max-num-seqs 2 \
187+
--kv-offloading-backend lmcache \
188+
--kv-offloading-size 1 \
189+
--disable-hybrid-kv-cache-manager
190+
```
191+
192+
- **Mitigation (silence)**: set `LMCACHE_LOG_LEVEL=CRITICAL`.
193+
- **Upstream issue**: [vLLM issue #30996](https://github.com/vllm-project/vllm/issues/30996).
194+
195+
#### vLLM log: `Found PROMETHEUS_MULTIPROC_DIR was set by user`
196+
197+
vLLM v1 uses `prometheus_client.multiprocess` and stores intermediate metric values in `PROMETHEUS_MULTIPROC_DIR`.
198+
199+
- If you **set `PROMETHEUS_MULTIPROC_DIR` yourself**, vLLM warns that the directory must be wiped between runs to avoid stale/incorrect metrics.
200+
- When running via Dynamo, the vLLM wrapper may set `PROMETHEUS_MULTIPROC_DIR` internally to a temporary directory to avoid vLLM cleanup issues. If you still see the warning, confirm you are not exporting `PROMETHEUS_MULTIPROC_DIR` in your shell or container environment.
201+
163202
## References and Additional Resources
164203

165204
- [LMCache Documentation](https://docs.lmcache.ai/index.html) - Comprehensive guide and API reference

docs/backends/vllm/prometheus.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,12 @@ python -m dynamo.vllm --model Qwen/Qwen3-0.6B --connector lmcache
129129
curl -s localhost:8081/metrics | grep "^lmcache:"
130130
```
131131

132+
### Troubleshooting
133+
134+
Troubleshooting LMCache-related metrics and logs (including `PrometheusLogger instance already created with different metadata` and `PROMETHEUS_MULTIPROC_DIR` warnings) is documented in:
135+
136+
- [LMCache Integration Guide](LMCache_Integration.md#troubleshooting)
137+
132138
**For complete LMCache configuration and metric details**, see:
133139
- [LMCache Integration Guide](LMCache_Integration.md) - Setup and configuration
134140
- [LMCache Observability Documentation](https://docs.lmcache.ai/production/observability/vllm_endpoint.html) - Complete metrics reference

0 commit comments

Comments
 (0)