Benchmark: NVIDIA DGX Spark (GB10) 121GB — 11.1+ tok/s full-k8 CUDA (PLD); was 2.4 / 3.33 chat

[VincentMarquez]

Machine:** NVIDIA DGX Spark (NVIDIA_DGX_Spark) · Ubuntu 24.04 · Linux 6.17.0-1008-nvidia aarch64
CPU: 10× Cortex-X925 @ 3.9 GHz + 10× Cortex-A725 @ 2.8 GHz (20 cores)
Memory: 121 GB unified (CPU/GPU shared)
GPU: NVIDIA GB10 (sm_121), driver 580.126.09
Disk: NVMe (/dev/nvme0n1p2), model on local ext4
Model: GLM-5.2 colibri int4 · int8 MTP heads (sizes match matey-0 clone: 3527131672 / 5366238584 / 1065950496)
Engine base: JustVugg/colibri @ a78a06f + local experimental patches (CACHE_ROUTE / TRACE_EXPERTS / CUDA unified path)

Disk (README protocol)

c/iobench · 19 MB × 64 reads · 8 threads · shard out-00099.safetensors:

mode	measured
buffered	4.25 GB/s
O_DIRECT	9.69 GB/s

How to read these numbers

Community table rows are usually stock routing (full top-8) with decode tok/s from the coli stats line after warm-up. This report has two tiers:

1. Tier A — full top-8, no routing change (config num_experts_per_tok=8)
2. Tier B — experimental CACHE_ROUTE (opt-in, not upstream default; ~14% expert substitution vs true top-8; cache-aware routing, arXiv:2412.00099 max-rank)

Also:

• Decode tok/s = generation window only (coli decode tok/s). Overall includes prefill — do not use overall for ranking.
• MTP=0 / DRAFT=0 for these speed cells.
• Chat used TEMP=0.7. Output on the primes prompt was the correct first-20 list every turn.
• Short context via :reset each turn (clears KV only; pin/LRU survive).

---

Tier A — full top-8, stock routing (TOPK unset, no CACHE_ROUTE)

Config: COLI_CUDA_UNIFIED=1 · CUDA_DENSE=1 · CUDA_GROUPED=1 · DIRECT=1 · PIPE=1 · large session LRU · learned AUTOPIN.

cell	protocol	decode tok/s	hit	notes
Timed ./glm PROFILE	WARM=160, NGEN=48, greedy TEMP=0	2.39	82%	best non-interactive full
Chat (warm session)	primes prompt, :reset each turn	~2.08	~82%	chat ladder peak before CA

Example footer (pre–CACHE_ROUTE):

└─ 39 tok · 2.08 decode tok/s · … · hit 82% · RSS ~76 GB
   miss ~107/tok · I/O ~8.5s · expert ~3.2s · attention ~3.9s

───

Tier B — experimental CACHE_ROUTE (opt-in; not stock default)

CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12
# keep true top-2; fill remaining 6 slots preferring pin∪LRU experts ranked in top-12

Sandbox predicted hit 0.81→0.955 and ~14% substitution. Live chat matches substitution and exceeds the hit target.

Interactive chat ladder

Prompt each turn:

List the first 20 prime numbers, one per line.

Protocol: one long-lived session · :reset before every turn · same prompt · full top-8 K · CACHE_ROUTE on.

┌──────┬─────┬──────────────┬─────┬──────┬──────┬──────────┐
│ turn │ tok │ decode tok/s │ hit │ swap │  I/O │ miss/tok │
├──────┼─────┼──────────────┼─────┼──────┼──────┼──────────┤
│    1 │  41 │         3.01 │ 95% │  15% │ 2.8s │     32.6 │
├──────┼─────┼──────────────┼─────┼──────┼──────┼──────────┤
│    2 │  39 │         3.16 │ 96% │  15% │ 2.1s │     25.1 │
├──────┼─────┼──────────────┼─────┼──────┼──────┼──────────┤
│    4 │  39 │         3.24 │ 96% │  14% │ 1.8s │     21.2 │
├──────┼─────┼──────────────┼─────┼──────┼──────┼──────────┤
│    6 │  39 │         3.29 │ 97% │  14% │ 1.6s │     18.4 │
├──────┼─────┼──────────────┼─────┼──────┼──────┼──────────┤
│ best │  41 │         3.33 │ 97% │  14% │ 1.5s │     18.0 │
└──────┴─────┴──────────────┴─────┴──────┴──────┴──────────┘

Best (Tier B): 3.33 decode tok/s · hit 97% · swap 14% · RSS 78.5 GB · wall ~18s for 41 tokens.

Verbatim footer:

└─ 41 tok · 3.33 decode tok/s · 2.27 overall · hit 97% · swap 14% · RSS 78.5 GB · 18s
   miss 18.0/tok (0.34 GB/tok) · MTP 1.00 tok/fw (0% accept) · I/O 1.5s · expert 3.2s · attention 4.1s
   CUDA 112.5% of expert calls (27675 calls) · 1.00 rows/expert
   cache admission 80 accepted · 660 scan-pollution rejects

Output quality on this prompt: correct first-20 primes every turn. Not a full ./coli bench quality gate — still pending before defaulting CACHE_ROUTE.

Timed one-shot (heavy warm, less favorable than multi-turn chat LRU):

48 tokens in 18.00s (2.67 tok/s decode) | expert hit rate 93.5% | swap 15.0%
PROFILE: expert-disk 3.852s | expert-matmul 3.833s | attention 6.155s | other 4.162s

[JustVugg]

Strong numbers, and CACHE_ROUTE is the interesting part — 2.4 → 3.33 tok/s (with a path to ~4.2) on a GB10 is a big jump, and it's a routing-side lever we don't have in-engine. Since it's a local patch, could you say more about what CACHE_ROUTE actually does? From the name I'd guess it caches/reuses the router's top-k decision across steps or tokens rather than recomputing — which would connect directly to the lookahead-routing work (routing is ~71.6% predictable one layer ahead, per the PILOT experiments) and to the profile-guided expert layout in #119. If it's what I think, it's a genuine contribution — a PR (even rough, behind an env flag like the rest) would let us A/B it against the lookahead path and land it properly. What's the mechanism, and does it change output (greedy-identical, or an accepted near-tie divergence)? This is exactly the kind of exclusive lever worth pulling into the engine.

[VincentMarquez]

@JustVugg — thanks. Happy to clarify the mechanism, share updated numbers (including a later engine stack on the same GB10), and yes: it's already behind env flags (default off). Happy to land a rough routing-only PR so you can A/B it against PILOT / #119.

What CACHE_ROUTE is not

It does not cache/reuse the previous token's top-k, and it does not skip the router. The router still scores all experts every layer every token.

What it is (max-rank cache-aware routing)

Inspired by arXiv:2412.00099 (max-rank style). Per sparse MoE layer, after the usual sigmoid router scores:

1. Rank experts by router score (same logits as stock).
2. Always take true top-ROUTE_J (default J=2), even if cold on disk.
3. Fill remaining K−J slots by preferring experts already resident (pin ∪ LRU) that still rank inside top-ROUTE_M (default M=12; we also use M=16).
4. If the window can't fill K, fall back to the true ranking for the rest.

Optional:

• ROUTE_ALPHA — scale gate mass of substituted experts before renorm (1 = off; some of our later cells used 0.5).
• ROUTE_AGREE=1 — footer overlap % vs true top-K + mean route KL (auto-on when CACHE_ROUTE=1).

So: routing-side, not I/O prediction. Complementary to PILOT (next-layer prefetch of weights; does not change expert IDs). CACHE_ROUTE can.

Lever	Acts on	Changes expert IDs?
PILOT	when cold experts load	no
CACHE_ROUTE	which experts are selected	yes (within top-M)
#119 layout	what ends up resident	no (placement)

Does it change output?

Not greedy-identical to stock top-8.
Substitution is real: ~14–22% swap of route slots vs true top-K; route_agree ~78–87% depending on M / warm. Same class as other near-tie MoE forks (README #100).

On a short primes prompt, multi-turn chat still produced the correct first-20 list every turn we checked (eyeball only — not a full ./coli bench gate). Keeping it opt-in / never default.

Env flags (already in the local engine)

# off by default
CACHE_ROUTE=0

# experimental bridge
CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12   # or ROUTE_M=16
# optional: ROUTE_ALPHA=0.5  ROUTE_AGREE=1

CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 COLI_MODEL=... ./coli chat

Footer / STAT: swap%, route_swaps/route_slots, route_agree, route_kl.

Numbers — NVIDIA DGX Spark (GB10, 121 GB unified)

All cells: full K=8 width (TOPK unset), decode tok/s, MTP off unless noted. Disk: 4.25 GB/s buffered · 9.69 GB/s O_DIRECT (iobench).

Tier A — stock routing (CACHE_ROUTE off)

cell	decode tok/s	hit	notes
Timed PROFILE (WARM=160, NGEN=48, greedy)	2.39	~82%	original post
Chat warm (primes, :reset)	~2.08	~82%	original post

Tier B — CACHE_ROUTE (early stack; original post)

Config: unified CUDA + dense/grouped + warm LRU; before fused-device decode.

cell	CR	decode tok/s	hit	swap	notes
Chat ladder best	J2 M12	3.33	97%	14%	multi-turn primes + :reset
Timed one-shot	J2 M12	2.67	93.5%	15%	less favorable than multi-turn chat

This is the pure routing-side jump you called out (~2.4 → 3.33).

Tier C — later stack on the same GB10 (updated)

Same host. Still full K=8 width. On top of CR we also shipped local engine work: GPU MLA (D3), fused S=1 decode (D4 / CUDA_FUSE), optional device-tier convert of hot experts under UNIFIED (CUDA_EXPERT_GB, host freed after upload), PILOT, often M=16 and ROUTE_ALPHA=0.5.

These are not “CACHE_ROUTE alone” — they show how far the box goes when residency + CUDA path + CR stack together.

cell	CR	decode tok/s	hit	swap / agree	notes
Strict, same later stack (CR off)	off	2.98	86.2%	—	fair A/B partner
CR M16, same later stack	J2 M16	5.14	94.0%	swap 19.6% · agree 80.4%	only CR differs from row above
CR M16, managed experts (no device-tier)	J2 M16	5.27	96.3%	—	fuse+MLA+pilot
CR M16 + device-tier 50GB	J2 M16	5.63–5.64	~96%	swap ~19% · agree ~81%	mid-window 6.17–6.21 on a confirm run
CR M16 + device-tier 50GB (repeat)	J2 M16	6.17	96.9%	swap 18.4% · agree 81.6%	full 48-tok timed window

Takeaway:

• CR alone (early stack): ~2.4 → 3.33 (chat).
• CR on/off with later stack held fixed: 2.98 → 5.14.
• Full later stack + CR: peaks ~5.6–6.2 decode tok/s on this GB10 (short context, warm, full K width).

Strict identity full-k8 without CR remains disk/residency limited (~3 tok/s class even with fuse). CR is still the main routing-side win; the 5–6 band also needs the CUDA/residency work.

PR shape

Routing-only, default CACHE_ROUTE=0, same style as PILOT=1:

• selection change in the MoE route path
• STAT/footer telemetry (swap, agree, kl)
• short flag note

We will not dump the full GB10 CUDA/fuse/device-tier stack into that PR so you can A/B CACHE_ROUTE vs PILOT / #119 cleanly. The later-stack numbers above are “what this machine can do”; the PR is the portable routing flag.

Happy to iterate on J/M/alpha defaults or a tiny unit test if you want a preferred shape before merge.

[ebootheee]

Reproduced your CACHE_ROUTE idea on x86 -- same box as #120/#215 (9950X3D2, dual 5090,Gen5 NVMe). No public patch on your fork that I could find, so I reimplemented it from your description: ROUTE_J true experts kept, remaining slots prefer cache-resident experts from the true top-M, substituted experts keep their own router logits and go through the standard norm_topk. ~40 lines, env-gated.

Numbers (1024-token windows, on top of my #215 config):

config	tok/s	hit	substitution
true top-8	2.2554	79%	--
ROUTE_J=2 ROUTE_M=12	3.09 / 3.04	92%	15.3%

So +42% here vs your +39% (2.39 -> 3.33) -- consistent, and that's over PCIe, no unified memory. Your substitution rate reproduces too (15.3% vs your 14%).

The part I wanted to add to the thread: a quality measurement. MMLU-200 A/B on the same build, same seeds, clean learned-cache state for both:

• true routing: 59.0%
• ROUTE_J=2 ROUTE_M=12: 62.0%

Before anyone reads that as "substitution helps" -- at n=200 the binomial noise is about ±3.5 points, so the honest conclusion is no detectable difference, not an improvement. But that's already a much stronger statement than we had: 15% expert ubstitution didn't measurably dent MMLU. A real claim wants the full benchmark (or at least --limit 1000); happy to run that overnight if there's interest.

@JustVugg this feels like it wants to be an official research flag rather than fork magic -- substitution rate printed on the result line, big warning in the README, and benchmark rows kept separate from true-routing numbers (mine above should NOT go in the main table). My ROUTE_J/ROUTE_M patch is small and I can fold it into the PR discussed in #215 if you want it.

[JustVugg]

This is one of the most useful reports in the tracker — thank you for the two-tier discipline (stock top-8 vs experimental CACHE_ROUTE) and for separating decode tok/s from overall. That's exactly how these should be read.

Two things here are landing directly in ongoing work:

1. O_DIRECT. Your 4.25 → 9.69 GB/s (buffered → O_DIRECT) on the NVMe is a striking 2.3× — it confirms the page-cache double-buffering tax on a fast drive. DIRECT=1 is worth flagging louder in the README for anyone on real NVMe; on slow/VHDX-backed disks it's less clear-cut (we measure ~232 MB/s on a Windows VHDX-backed WSL setup, disk-bound long before O_DIRECT matters), so it's genuinely a per-tier lever, as you framed it.

2. CACHE_ROUTE is the interesting one. 2.39 → 3.33 decode at ~14% expert substitution, quality holding on the primes prompt, is a strong result for the max-rank cache-aware idea (arXiv:2412.00099). We've been circling the same problem from the low-RAM end: EXPERT_BUDGET (blind drop by gate weight) speeds up but collapses coherence on reasoning prompts, whereas CACHE_ROUTE's substitution toward already-resident experts preserves far more — it's the principled middle path. If you can share a quality A/B (greedy-token agreement vs true top-8 across a small reasoning set, not just the primes list), that would help justify tuning ROUTE_J/ROUTE_M defaults, and it connects to @jeswr's expert-specialization work in #207 (which experts are safe to substitute).

If you're up for it, please repost the benchmark in Discussions → Show and tell (#208) so the DGX Spark row sits alongside the other datapoints. And thank you for pushing CACHE_ROUTE — it may be the lever that makes low-RAM hosts usable without the quality cliff.

[FiveTechSoft]

+1 — same GB10 silicon, MiMo-V2.5 results + what moved the needle

Same chip here: DGX Spark, 121 GB unified (CPU<->GPU), GB10 sm_121, CUDA 13.0 / 580.159.03, 20x ARM (10x X925 + 10x A725). We run MiMo-V2.5 (311B MoE, ~15B active, 256 experts/layer top-8) under peng-mimo (the colibri fork adapted to MiMo). Our numbers line up with yours and add a data point on a smaller-active model.

Decode speed (MiMo-V2.5 int4, 152 GB container — experts are 93% of the weight):

config	tok/s
CPU only (COLI_CUDA=0)	0.85
GPU, expert cache auto	1.88
GPU, CUDA_EXPERT_GB=80	3.14
GPU, CUDA_EXPERT_GB=100 + CUDA_DENSE=1	4.82

So on identical hardware we're already at 4.82 vs your 2.4 full / 3.33 CACHE_ROUTE — mostly because MiMo's active set is smaller and the dense path is resident on GPU (CUDA_DENSE=1).

Technique that got us there

• int4 experts + int8 dense (operating point ./mimo <cap> 4 8); no O_DIRECT yet.
• autopin from a persistent .coli_usage (~ PIN=auto, PIN=auto: seed the pin from the live usage history #301).
• GPU expert-cache budget (CUDA_EXPERT_GB, ~ EXPERT_BUDGET); 100 GB -> 4.82.
• Incoherent-output root cause = your Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225. We hit the exact failure (fluent but off-topic). In our case it was a per-rank scale-grid bug in the fused-qkv converter (NR=4: the fp8 scale_inv has 108 rows/rank, not 106), so ranks 1-3 were corrupt while rank 0 / o_proj / all experts stayed bit-identical. Caught it with a bit-exact F32+int4 oracle and a regression guard (c/tools/verify_mimo_qkv.py). The grouped-int4 gs=64 fix (Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 / PR cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298) is the real cure — evaluating it now.
• Disk pole. Our effective throughput under the engine's random small expert reads is only ~1.2 GB/s (your 4.25 buffered / 9.69 O_DIRECT makes me want O_DIRECT + io_uring, [Feature]: (Performance) Implement non-blocking Async I/O for expert loading (io_uring on Linux, Metal File I/O on macOS) #251, badly). The other big lever we're about to test: prune to UNION179 (179/256 experts) -> 117 GB store, which essentially fits the 121 GB unified -> disk pole gone. That alone should clear your 4.2 target.

Levers we're pulling from this thread: cloxcache (#223), co-activation re-layout (#119), i8mm/SMMLA (#217), O_DIRECT (#161/#251).

Containers + engine: github.com/FiveTechSoft/peng-mimo; HF fivetech/MiMo-V2.5-colibri-peng-int4 (152 GB) and -int4-zstd (109 GB). Happy to share the MiMo adapter, the qkv scale-grid fix, or the prune layout if useful.

[mohamedmastouri2000-boop]

Datapoint: RTX 5080 (Blackwell, sm_120) 16 GB — g64 container fully GPU-resident, ~1.0 tok/s decode at full K=8

Host: RTX 5080 (sm_120, 17.1 GB usable) / Core Ultra 9 285K / 128 GB DDR5 / Windows 11 native, CUDA 13.3. Container: public mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp (grouped int4 gs=64 + int8 MTP). Engine built with the fmt=4 CUDA path from #298 (without it, grouped-int4 silently runs on CPU — see my validation there).

Config: COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 CUDA_EXPERT_GB=4. Full K=8 width (TOPK unset). MTP inactive (draft=0). Greedy, NGEN=48.

Residency: 12.48 GB dense (952 tensors) + 109 experts (2.31 GB) live in VRAM → ~14.8 GB used. .coli_usage-primed placement (109 VRAM + ~2064 RAM warm).

pass	decode tok/s	expert hit	experts/token	expert calls from VRAM
cold (usage-primed)	0.99	83.7%	587.5	2837
warm	1.00	84.1%	587.5	3000

Prefill is disk-bound: 19 tokens in ~33 s. Decode holds ~1.0 tok/s — the ~84% expert hit means the residency stack (dense + 109-expert VRAM tier) is carrying most calls, but the ~16% cold-expert misses still stream from NVMe per token, which is the current ceiling on this box. No TDR, no fallbacks, coherent output.

Reads as a mid-Blackwell counterpart to the GB10/5090 entries: same full-K=8 discipline, dense + partial expert tier resident, decode gated by NVMe streaming of the cold-expert tail rather than compute. Happy to add an EXPERT_GB sweep (dense-only vs larger VRAM tier), MTP-on numbers, or a longer-context profile if useful for the tracker.

[VincentMarquez]

Update — DGX Spark (GB10) full top-8 campaign toward 30 tok/s

Still on the same box: NVIDIA DGX Spark · GB10 · 121 GB UMA · aarch64 · local int4 + int8 MTP. Goal for this thread remains 30 decode tok/s at full top-8 quality (no TOPK prune). We have not hit 30 yet; this is a progress report + open work, not a leaderboard claim of 30.

Headline numbers (timed ./glm, full K=8, TEMP=0)

tier	mid [t=16]	overall decode	tok/fw	notes
S=1 fuse, full k8 + CR	~6.7–7.3	~6.5–6.9	1.0	hot experts (~99% hit), short-context warm
Multi-S + free PLD + CR	~9.5 peak	~8.5–8.6 peak	up to ~4.0	best full-k8 e2e so far (high variance)
S=1 TOPK=1 (quality reduced)	~14	~13	1.0	ceiling check only — not full-k8

Earlier public points on this issue (~2.4 full / ~3.3 CACHE_ROUTE chat) are superseded on the timed decode path by the fused S=1 + residency stack. Chat vs timed still differ; we keep reporting decode tok/s from the timed window after warm.

What we implemented / measured (local C engine work)

All of this is still experimental / local relative to stock upstream unless noted — not claiming a merged PR here:

• Device-resident S=1 fused decode (CUDA_FUSE + managed KV + GPU MLA): attention/experts stay on the UMA path; PROFILE still folds some expert drain into the attention bucket.
• Multi-S verify path (S=2..4) for MTP/PLD: batch MLA + device routers + per-row FusedMoe (or opt-in union). Residual-on-GPU path is required; residual D2H every layer was a multi-S e2e killer.
• CACHE_ROUTE still the residency lever (J/M/α cells). Sacred / no-swap modes preserve logits better for accept but thrash more without CR fill.
• Free drafts first: PLD / n-gram (PLD_FIRST / FORCE_PLD) before the MTP head — on list-like prompts this is what drove tok/fw ≈ 3–4 when it hit.
• Warm n-gram corpus after WARM_RESET_KV (keep warm tokens for PLD without long KV) — works as plumbing; accept after reset is still weaker than long-context PLD.
• MoE-Spec-style EXPERT_BUDGET, MTP_MARGIN early-stop, adaptive draft depth, MTP_DRAFT_TOPK, DRAFT capped at 3 under multi-S fuse (S≤4).
• Physics takeaway on this silicon: full-k8 S=1 is roughly a ~7 tok/s wall (MLA bandwidth + top-8 experts). 30 needs multi-S verify cost ≲ ~1.1× S=1 with sustained AL ≳ 4, or a much faster S=1 baseline. Today best net is ~8.5–9.5 when PLD multi-S pays — about ~3× short of 30.

Still working on (toward 30 full-k8)

1. Cheaper multi-S verify — expert path still too heavy vs S=1 (shared CUDA expert scratch forces serial S positions; true multi-stream needs per-stream scratch).
2. Sustained high AL without long-context tax — long warm KV helps PLD (AL up to ~4) but inflates attention; short-ctx + corpus is not yet a net win.
3. MTP accept vs draft cost — full-k8 MTP can accept well but draft “other” can erase the win; free PLD is the reliable multi-S fuel so far.
4. Honest quality separation — full top-8 vs CACHE_ROUTE swap% vs TOPK=1 ceiling will stay labeled separately (same discipline as the original Benchmark: NVIDIA DGX Spark (GB10) 121GB — 11.1+ tok/s full-k8 CUDA (PLD); was 2.4 / 3.33 chat #161 two-tier report).

Goal (unchanged)

30 decode tok/s on full top-8 (K=8) on this GB10 / 121 GB UMA host, without silently collapsing to TOPK=1.

Happy to take protocol notes if the project wants a standard “Spark full-k8 timed” row for the community table. More numbers as the multi-S cost comes down.

— continuing on the local C stack; no claim that 30 is done.

[mohamedmastouri2000-boop]

Follow-up datapoint: low-RAM sweep on the same RTX 5080 box — at 25 GB the compute tier stops mattering

Same host and protocol as my earlier entry (RTX 5080 sm_120 / Core Ultra 9 285K / 128 GB DDR5 / Windows 11, g64 grouped-int4 container, full K=8, greedy, 64-token decode, PIPE=1 DIRECT=1, dev @ 68ac9ff). I ran a controlled RAM sweep — full budget (~121 GB auto) vs RAM_GB=25 — across three compute tiers, both drives.

RAM budget	GPU (dense+experts VRAM)	GPU-experts-only	CPU (--gpu none)
full (~121 GB)	0.89 tok/s · hit 84%	0.72 · hit 86%	0.62 · hit 77%
25 GB	0.37 tok/s · hit 32%	0.38 · hit 33%	0.31 · hit 16%

What I take from it for the low-RAM-hosts question:

• The RAM cliff dwarfs everything else. Halving nothing but RAM costs ~55–60% in every tier. At 25 GB the engine reports cap lowered 8->2 and the warm tier collapses — most expert bytes stream from NVMe per token (~5.4 GB/s sustained, measured live).
• At 25 GB, GPU ≈ CPU (0.37 vs 0.38 vs 0.31). Once decode is streaming-bound, the matmul device is irrelevant. A "small" host with a big GPU is still a small host.
• Drive choice is a rounding error by comparison: the same sweep from the second (OS) NVMe was consistently just 3–6% slower; a two-drive split of the shards changed nothing at full RAM (0.92 vs 0.89, within noise).
• One asymmetry worth knowing when reading CPU rows anywhere in the tracker: CPU mode auto-enables MTP (draft=3, ~2.4–3.0 accepted tok/forward here) while CUDA has it off by default — the 0.62/0.31 CPU numbers are speculation-assisted.

This lines up with the direction already in this thread: for hosts under the warm-set size, more bandwidth or more compute barely move the needle — routing-side levers like CACHE_ROUTE (which specifically raises hit rate) look like the only real escape. Happy to run a CACHE_ROUTE × 25 GB cell on this box once the routing-only PR lands upstream, if a Blackwell/Windows datapoint helps.

[mohamedmastouri2000-boop]

CACHE_ROUTE on Blackwell/Windows — first sm_120 datapoint: +34% at M16 (matches the GB10/x86 numbers), +44% at M24; plus a CR × MTP interaction finding

Same box/protocol as my earlier entries (RTX 5080 sm_120 / 128 GB DDR5 / Windows 11, g64 grouped-int4, CUDA_DENSE=1 + 4 GB expert tier, PIPE=1 DIRECT=1, full K=8, greedy, 64-token decode). Engine: current dev @ 4aca059. Two-tier discipline as established in this thread — all CR rows are the experimental opt-in, labeled with swap/agree, not comparable to stock rows.

Tier A — stock routing (same session): 0.82 tok/s @ 74.5% hit.

Tier B — CACHE_ROUTE ladder:

cell	decode tok/s	hit	swap	route_agree
J2 M12	0.83	87.7%	15.2%	84.8%
J2 M16 α0.5	1.10	91.3%	22.4%	77.6%
J2 M24 α0.5	1.18	94.4%	26.9%	73.1%

So the lever reproduces on a third architecture/OS: +34% at M16, right in line with the GB10 (+39%) and 9950X3D/x86 (+42%) datapoints. M24 extends the sweep past anything published — +44%, but swap rises to ~27% (agree 73%), so quality scrutiny should scale with it; I'd call M16 the defensible operating point and M24 "needs a quality A/B before anyone uses it." Prefill also benefits big (30.5s → 16.5s: fewer cold reads).

CR × MTP interaction (COLI_CUDA_MTP=1, post-#470): CR M16 + MTP = 0.77 tok/s — still net-negative, but MTP acceptance jumped to 67% (42/63), 3.05 tok/forward — the highest CUDA acceptance I've seen reported, up from 49% without CR. Reading: CR substitution steers toward experts the MTP path also finds warm, so drafts verify better — but the S=4 union still drags hit down (91→87%) and the streaming cost still wins at this residency. Sharpens the crossover picture: even 94% hit isn't enough; MTP-under-CUDA really does want ~99% / hot-set territory.

Note for anyone comparing against my older rows: this session's stock baseline came in lower than my previous 0.95 (74.5% vs 86% hit, LRU share collapsed) — cause not yet isolated (investigating; possibly cache-state drift vs an engine change, will follow up separately rather than pollute this thread). The CR deltas above are all same-session, same-binary.

[mohamedmastouri2000-boop]

Update to my CR ladder above: those numbers were measured on a binary carrying the #474 regression (found in #490, fixed by #497). On current dev (#497 + #477 + #495) everything moved up — new Blackwell/Windows record: 1.27 tok/s full-K=8.

cell	regressed binary (above)	current dev
stock routing	0.82 (hit 74.5%)	0.89 (hit 79.4%) — 0.96 with PILOT_EVICT_GUARD=0
CR J3 M16 α0.5 (balanced)	1.10	1.18 · swap 17.9% · agree 82.1% · KL 2.93
CR J2 M24 α0.5 (speed)	1.18	1.27 · hit 96.4% · swap 23.3%

Two things worth noting for CR specifically: a healthy LRU disproportionately benefits routing-assisted configs (the regression's biggest victim was CR, not stock — more warm residents means better-ranked substitutes), and the fidelity metrics improved alongside speed (J3/M16's KL 3.42 → 2.93). Full config for reproduction: CUDA_EXPERT_GB=4 CUDA_RESERVE_GB=1 PIPE=1 DIRECT=1 CACHE_ROUTE=1 ROUTE_J=3 ROUTE_M=16 ROUTE_ALPHA=0.5 (add ROUTE_J=2 ROUTE_M=24 for the speed point).

Cumulative for the tracker's 16 GB-consumer-Blackwell row: 0.67 (defaults) → 1.27 (current dev + flags) over three days, all on stock upstream code — no local patches.

[VincentMarquez]

Update 2026-07-23 — full top-8 decode >11 tok/s on DGX Spark (GB10)

New controlled decode runs on the same machine. Not stock ./coli chat — experimental CUDA path (fused S=1 + GPU MLA + large expert VRAM tier + PLD). Reported carefully so it does not get mixed with the earlier 2.4 / 3.33 chat ladder.

Headline (full top-8, timed decode only)

cell	decode tok/s	mid t=16	AL	expert hit
k8maxal2	11.12	10.50	3.37	99.9%
k8w16_maxal_seed	11.17	11.27	3.46	~100%
k8idot_union	11.19	—	3.90	99.9%
k8push30	10.94–10.98	11.71	4.00	~100%

Protocol (honest)

• Full top-8 (experts/token ≈ 600). CACHE_ROUTE J=1 M=32.
• FORCE_PLD n-gram drafts (DRAFT=3), MTP off. AL = tokens/forward.
• Short synthetic prompt (~40 tokens), WARM discarded, timed NGEN=128–160.
• CUDA: unified + dense + grouped + GPU MLA + CUDA_FUSE + ~64 GB expert tier.

Verbatim 11.12

128 tokens in 11.51s (11.12 tok/s decode) | expert hit rate 99.9% | RSS 11.40 GB
experts loaded/token: 600.0 (per-layer 8.00 across 75)
speculation: 3.37 tokens/forward | MTP acceptance 0%
PROFILE: expert-disk 0.466s | expert-matmul 0.137s | attention 9.275s

Physics

~80% of timed decode is MLA/attention once experts are pinned. Path to 30 still needs ~2–3× faster MLA.

Lab record repo update staged at: https://github.com/VincentMarquez/glm52-gb10-colibri (push pending if not already live)