You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/backends/vllm/LMCache_Integration.md
+39Lines changed: 39 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -160,6 +160,45 @@ When LMCache is enabled with `--connector lmcache` and `DYN_SYSTEM_PORT` is set,
160
160
161
161
For detailed information on LMCache metrics, including the complete list of available metrics and how to access them, see the **[LMCache Metrics section](prometheus.md#lmcache-metrics)** in the vLLM Prometheus Metrics Guide.
162
162
163
+
### Troubleshooting
164
+
165
+
#### LMCache log: `PrometheusLogger instance already created with different metadata`
166
+
167
+
You may see an error like:
168
+
169
+
```text
170
+
LMCache ERROR: PrometheusLogger instance already created with different metadata. This should not happen except in test
171
+
```
172
+
173
+
**Version note**: We reproduced this behavior with **vLLM v0.12.0**. We have not reproduced it with **vLLM v0.11.0**, so it may be specific to (or introduced in) v0.12.0.
174
+
175
+
This is emitted by LMCache when the LMCache connector is initialized more than once in the same process (for example, once for a `WORKER` role and later for a `SCHEDULER` role). LMCache uses a process-global singleton for its Prometheus logger, so the second initialization can log this warning if its metadata differs.
176
+
177
+
-**Impact**: This is a log-only error; in our testing it does not prevent vLLM/Dynamo from serving requests. If you care about LMCache metric labels, be aware the logger singleton uses the first-seen metadata.
178
+
-**Repro without Dynamo** (vLLM v0.12.0):
179
+
180
+
```bash
181
+
vllm serve Qwen/Qwen3-0.6B \
182
+
--host 127.0.0.1 --port 18000 \
183
+
--gpu-memory-utilization 0.24 \
184
+
--enforce-eager \
185
+
--no-enable-prefix-caching \
186
+
--max-num-seqs 2 \
187
+
--kv-offloading-backend lmcache \
188
+
--kv-offloading-size 1 \
189
+
--disable-hybrid-kv-cache-manager
190
+
```
191
+
192
+
-**Mitigation (silence)**: set `LMCACHE_LOG_LEVEL=CRITICAL`.
#### vLLM log: `Found PROMETHEUS_MULTIPROC_DIR was set by user`
196
+
197
+
vLLM v1 uses `prometheus_client.multiprocess` and stores intermediate metric values in `PROMETHEUS_MULTIPROC_DIR`.
198
+
199
+
- If you **set `PROMETHEUS_MULTIPROC_DIR` yourself**, vLLM warns that the directory must be wiped between runs to avoid stale/incorrect metrics.
200
+
- When running via Dynamo, the vLLM wrapper may set `PROMETHEUS_MULTIPROC_DIR` internally to a temporary directory to avoid vLLM cleanup issues. If you still see the warning, confirm you are not exporting `PROMETHEUS_MULTIPROC_DIR` in your shell or container environment.
201
+
163
202
## References and Additional Resources
164
203
165
204
-[LMCache Documentation](https://docs.lmcache.ai/index.html) - Comprehensive guide and API reference
Troubleshooting LMCache-related metrics and logs (including `PrometheusLogger instance already created with different metadata` and `PROMETHEUS_MULTIPROC_DIR` warnings) is documented in:
0 commit comments