🚀 The feature, motivation and pitch
The v1 KV cache block manager currently supports two tiers — GPU (primary) and host
(secondary). We'd like to extend it with a third, disk-backed tier: blocks evicted from
the host tier spill to local disk and are reused on a later prefix hit instead of recomputed,
extending KV reuse well beyond DRAM.
We'd also like to add a per-request retention period — a request can ask that its KV cache
stay reusable for a given amount of time. The engine has no such control today.
We've implemented both for the v1 manager and validated them under production traffic at
DeepInfra. A PR is ready and will be opened shortly.
Alternatives
KVCacheManagerv2 does have a disk tier, but in our production testing at DeepInfra it
cannot sustain production traffic (it has to run with the overlap scheduler disabled, and
even then it can't keep up), and it has no per-request retention. So it does not cover the
default v1 path, nor the retention capability we need.
Additional context
Implementation (C++ + Python, with a disk-tier unit-test suite and benchmarks) is ready; we
will open the PR shortly.
🚀 The feature, motivation and pitch
The v1 KV cache block manager currently supports two tiers — GPU (primary) and host
(secondary). We'd like to extend it with a third, disk-backed tier: blocks evicted from
the host tier spill to local disk and are reused on a later prefix hit instead of recomputed,
extending KV reuse well beyond DRAM.
We'd also like to add a per-request retention period — a request can ask that its KV cache
stay reusable for a given amount of time. The engine has no such control today.
We've implemented both for the v1 manager and validated them under production traffic at
DeepInfra. A PR is ready and will be opened shortly.
Alternatives
KVCacheManagerv2 does have a disk tier, but in our production testing at DeepInfra it
cannot sustain production traffic (it has to run with the overlap scheduler disabled, and
even then it can't keep up), and it has no per-request retention. So it does not cover the
default v1 path, nor the retention capability we need.
Additional context
Implementation (C++ + Python, with a disk-tier unit-test suite and benchmarks) is ready; we
will open the PR shortly.