cache-warming-strategy-bench

Benchmark of KV-cache warming strategies for LLM serving.

Measures how quickly each strategy brings a cold server to steady-state hit rate and how much TTFT improves for the first requests after a scale-up event.

Why This Exists

Three prior projects pointed at this gap:

• model-cold-start-bench showed that weight pre-loading reduces startup to under 5s — but the KV cache is still cold.
• system-prompt-cache-bench showed that a prefix cache hit saves 99% of prefill work.
• radix-attention-sim showed that concurrent sharing depends on a pre-populated cache.

After scale-up, a cold server pays full prefill cost for 30–68 seconds before reaching steady-state hit rate. This benchmark asks: can a deliberate warming strategy compress that window?

Strategies

Strategy	Description
cold_start	No warming. Baseline.
top_k_prompts	Pre-compute top-K most popular prompts. Stops at K regardless of remaining budget.
top_k_adaptive	Pre-compute popular prompts in rank order until the warm cache budget is full.
top_k_adaptive_reserve	Same as adaptive but fills only 80% of budget, reserving 20% for online learning during live traffic.
replay_last_n	Install prompts seen in the last N requests from the previous server.
progressive_warming	Install prompts opportunistically on first access during live traffic.

Key Results

1) top_k_adaptive_reserve is the best practical strategy

Averaged across all models and workloads:

Strategy	Mean TTFT	Hit rate	TTFT reduction	Budget used
cold_start	50.3 ms	0.000	—	0%
top_k_prompts	12.3 ms	0.772	75.5%	63%
top_k_adaptive	11.6 ms	0.821	76.9%	92%
top_k_adaptive_reserve	11.5 ms	0.824	77.2%	97%
replay_last_n	13.4 ms	0.774	73.4%	—
progressive_warming	13.9 ms	0.776	72.4%	—

Reserving 20% of the warm cache for online learning during live traffic:

• Matches or beats top_k_adaptive in 7 of 8 scenarios
• Beats progressive_warming in 4 of 8 scenarios
• top_k_adaptive never beats reserve by more than 0.5 ms (0/8 scenarios)

2) Concentrated workloads reach 90%+ TTFT reduction

Best headline cases (Qwen2-0.5B):

Workload	Strategy	Cold TTFT	Warmed TTFT	Reduction	Hit rate
autoscale_spike	progressive_warming	51.1 ms	4.0 ms	92.1%	0.949
enterprise_hotset	top_k_adaptive	49.0 ms	4.1 ms	91.7%	0.956
mixed_prompts	top_k_adaptive	26.5 ms	7.2 ms	72.8%	0.882
diverse_cold	top_k_adaptive_reserve	14.9 ms	6.9 ms	53.5%	0.792

3) Fixed top_k fails in diverse workloads — adaptive strategies fix it

In diverse_cold (50 system prompts, low Zipf concentration), top_k_prompts uses only 24–40% of the warm cache budget and achieves 52% hit rate. Adaptive strategies fill the budget and improve substantially:

Model	Strategy	Mean TTFT	Hit rate	Time to steady
qwen2_0.5b	cold_start	14.9 ms	0.000	68388 ms
qwen2_0.5b	progressive_warming	7.5 ms	0.740	3249 ms
qwen2_0.5b	replay_last_n	7.8 ms	0.708	3777 ms
qwen2_0.5b	top_k_adaptive	6.9 ms	0.776	3249 ms
qwen2_0.5b	top_k_adaptive_reserve	6.9 ms	0.792	3249 ms
qwen2_0.5b	top_k_prompts	9.5 ms	0.524	10354 ms
qwen2_1.5b	cold_start	27.3 ms	0.000	68442 ms
qwen2_1.5b	progressive_warming	18.7 ms	0.465	3249 ms
qwen2_1.5b	replay_last_n	18.0 ms	0.491	8168 ms
qwen2_1.5b	top_k_adaptive	15.5 ms	0.618	3846 ms
qwen2_1.5b	top_k_adaptive_reserve	15.8 ms	0.605	3846 ms
qwen2_1.5b	top_k_prompts	17.5 ms	0.524	10354 ms

4) Cold start degrades TTFT for 33–68 seconds

Without warming, time to reach stable hit rate ranges from 33,000 ms to 68,000 ms. All adaptive warming strategies compress that to 1,000–10,000 ms.

Time to steady state — top_k_adaptive_reserve vs cold_start:

Workload	Cold baseline	Reserve strategy	Gap
qwen2_0.5b / autoscale_spike	33,368 ms	1,042 ms	-32,327 ms
qwen2_0.5b / diverse_cold	68,388 ms	3,249 ms	-65,139 ms
qwen2_0.5b / enterprise_hotset	50,867 ms	2,144 ms	-48,723 ms
qwen2_0.5b / mixed_prompts	63,286 ms	2,418 ms	-60,869 ms
qwen2_1.5b / autoscale_spike	33,465 ms	1,042 ms	-32,423 ms
qwen2_1.5b / diverse_cold	68,442 ms	3,846 ms	-64,596 ms
qwen2_1.5b / enterprise_hotset	50,963 ms	2,144 ms	-48,819 ms
qwen2_1.5b / mixed_prompts	63,380 ms	2,418 ms	-60,962 ms

5) Warming compute cost is always under 270 ms

One-time pre-warming compute investment across all scenarios:

• top_k_adaptive: 238.3 ms average
• top_k_adaptive_reserve: 250.5 ms average
• progressive_warming: 249.0 ms average (amortized during live traffic)

Zero SLO violations in any warming strategy across all tested scenarios.

Practical Recommendations

Situation	Recommended strategy
Best default for production	top_k_adaptive_reserve
Prompt distribution unknown or diverse	progressive_warming
Previous server request log available	replay_last_n (N ≥ 50)
Avoid	top_k_prompts with fixed K — always superseded by adaptive

Experimental Scope

• Models: 2 (Qwen2-0.5B, Qwen2-1.5B)
• Workloads: 4 (enterprise_hotset, mixed_prompts, diverse_cold, autoscale_spike)
• Strategies: 6
• Seeds: 3
• Raw simulation runs: 264
• Aggregated scenario rows: 48

Quick Start

git clone https://github.com/JohnScheuer/cache-warming-strategy-bench
cd cache-warming-strategy-bench

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python run.py

Runtime: approximately 1-2 minutes. Hardware: CPU-only.

Output Files

results/
  summary.csv
  summary_agg.csv
  recommendations.csv

plots/
  01_ttft_qwen2_0.5b.png / 01_ttft_qwen2_1.5b.png
  02_hit_rate_qwen2_0.5b.png / 02_hit_rate_qwen2_1.5b.png
  03_time_to_steady_qwen2_0.5b.png / 03_time_to_steady_qwen2_1.5b.png
  04_warming_cost_vs_ttft.png

Documentation

• DESIGN.md — simulator design, strategy definitions, metric descriptions
• SUMMARY.txt — plain-text findings with all numbers
• LICENSE — MIT License

Related Projects

• model-cold-start-bench
• system-prompt-cache-bench
• radix-attention-sim
• slo-aware-autoscaling-sim

License

MIT License — Copyright (c) 2026 João Felipe De Souza

Author

João Felipe De Souza