Benchmark: GLM-5.2 on Ryzen 7 9850X3D + RTX 5090, native Windows (0.89 tok/s median)

[Appie-NL]

Machine datapoint

component	detail
CPU	AMD Ryzen 7 9850X3D, 8 cores / 16 threads
RAM	64 GB DDR5 4800MT/s
DISK	2TB WD_Black SN8100
GPU	NVIDIA GeForce RTX 5090, 32607 MiB, driver 591.86
OS	native Windows 11
model	GLM-5.2 Colibri int4
Colibri commit	4ef9a9920dd4290b4ac26f74e73913fd18379bc9
build	native Windows CUDA DLL, sm_120

Configuration

COLI_CUDA=1
CUDA_DENSE=1
CUDA_EXPERT_GB=auto
CUDA_RESERVE_GB=3
COLI_CUDA_PIPE=2
COLI_CUDA_ATTN=1
PIN=auto
PIN_GB=20
RAM_GB=52
CAP_RAISE=1
DRAFT=0
PIPE=1
PIPE_WORKERS=8
DIRECT=1
PILOT=1
PILOT_REAL=1
PILOT_WORKERS=4
cap=8
TEMP=0
NGEN=32

Fixed prompt: Continue this story: The lighthouse keeper climbed the stairs and saw something impossible in the fog.

The same fixed prompt was run three times. Existing .coli_usage history was retained, so these are warm, usage-trained runs rather than clean-install cold runs.

Decode results

run	prefill (21 tokens)	decode (32 tokens)	throughput	expert hit rate	RSS
1	18.77 s	35.57 s	0.90 tok/s	38.1%	43.10 GB
2	18.93 s	36.15 s	0.89 tok/s	38.1%	43.10 GB
3	18.60 s	36.91 s	0.87 tok/s	38.2%	43.10 GB
median	18.77 s	36.15 s	0.89 tok/s	38.1%	43.10 GB

Additional runtime details:

• CUDA resident set: 20.32 GB VRAM
• CUDA expert tier: 478 resident experts / 10.15 GB
• MTP compiled active, but DRAFT=0; measured 1.00-1.03 tokens/forward
• First run reported 581.2 experts loaded per token

Disk

iobench out-00069.safetensors 19 64 8 <mode>:

mode	throughput
buffered, 8 threads	4.80 GB/s
O_DIRECT, 8 threads	10.64 GB/s

Thread scaling with 128 reads: 1T 9.57, 2T 9.90, 4T 9.85, 8T 10.65, 12T 10.72, 16T 9.57 GB/s.

Runner compatibility note

The current c/tools/datapoint.py could not directly produce this GLM datapoint on native Windows. It invokes the older engine interface with positional cap and bits arguments, expects the older resident weights loaded in ... | RSS after load log line, and reports Windows RAM as unknown. The GLM engine instead takes the prompt through PROMPT, uses NGEN/MAX_NEW, and prints loaded in ... | resident dense: .... These measurements therefore use the documented fixed 32-token protocol directly against colibri.exe.

[JustVugg]

Excellent datapoint — three runs, fixed prompt, median, config disclosed: this is the shape we wish every benchmark had. And the number is coherent: at 38.1% hit rate you're loading ~390 experts/token ≈ 7 GB/token from disk, and at 1.12 s/token that's ~6.5 GB/s effective random read — your SN9100 is near its practical ceiling. The box is disk-bound, and the config is doing what it says.

Which means the two biggest levers are not kernels:

1. DRAFT=0 → try DRAFT=3. MTP is compiled and idle in your runs (1.00–1.03 tokens/forward). On a disk-bound workload the draft multiplies tokens per forward almost linearly — this is the single largest knob you have, and it costs one env change.
2. cap=8 looks starved. 8 cache slots per layer is tiny for 64 GB of RAM, and your 38% hit rate + RSS at 43 GB with RAM_GB=52 (9 GB unused) both point the same way. Check the startup log for the cap auto-raise line; if CAP_RAISE didn't engage, set a much larger cap explicitly and watch the hit rate move.

Also worth trying: current dev. Since your commit (4ef9a99) the expert matmul path gained a layer-level activation-quantization hoist (#1071 series), a plane-nibble int4 layout + register tile (#1079/#1088 — 1.5–3× on the expert matmul where compute binds), and a fused expert pass (#1093). On your box decode is disk-bound so those mostly move prefill (your 18.8 s for 21 tokens should drop measurably); decode moves with the two levers above. If your container is int4-per-row the kernel gains apply as-is; if it's int4-gs64, #1094 (in CI now) adds an opt-in IDOT_GS=1 fast path for it.

If you re-run the same 3-run protocol with DRAFT=3 + a raised cap on current dev and post the same table, that before/after would be genuinely valuable — same methodology, same prompt, so the deltas are clean.

[JustVugg]

Correction to my own advice above — please don't run DRAFT=3. I checked the engine's own measured guidance after posting, and I got it wrong twice:

1. Depth 3 is not the setting. A GLM-5.2 744B sweep in-tree (DRAFT=0/1/2/3, streaming and fully-resident) found depth 1 is the only depth that pays: acceptance ~85% at depth 1 vs ~44–62% at 2–3, and every extra draft token both costs verify compute and — when streaming — faults experts that evict the LRU working set. Depth 1 was fastest in every measured configuration.
2. On CUDA, MTP is off by default for a documented reason (MTP acceptance regression: 33% -> 4% (same machine/config/int8 head) between pre-#68 main and current dev #163): cold streamed experts run on the CPU, where the S==1 fused-pair kernel and the S≥2 IDOT kernel diverge in FP accumulation order, collapsing draft acceptance. Your config is CUDA, so DRAFT=n alone would not even have enabled it — COLI_CUDA_MTP=1 is the opt-in, and MTP acceptance regression: 33% -> 4% (same machine/config/int8 head) between pre-#68 main and current dev #163 reports acceptance still reaching 30–50% despite the mismatch, which may or may not pay on your box.

So the honest version of lever 1 is: COLI_CUDA_MTP=1 DRAFT=1, as an experiment worth one run — not a recommendation. Lever 2 (the starved cap=8 against 64 GB of RAM, 38% hit rate, 9 GB unused) stands unchanged and is, I now think, clearly the bigger one for your machine.

Sorry for the noise — better a correction than a wasted evening.

[Appie-NL]

Follow-up measurements on current dev are complete. I preserved the original build and built commit 5c822f71683c2744e115298d8fa2e4aa9cf74d37 separately as native Windows CUDA/sm_120.

One hardware clarification: the 64 GB kit is Kingston Fury Beast RGB KF560C30BBAK2-64, now running with XMP enabled at DDR5-6000 CL30.

Same model, fixed prompt and 32-token greedy protocol as the original datapoint. Common placement stayed CUDA_EXPERT_GB=auto CUDA_RESERVE_GB=3 PIN=auto PIN_GB=20 DRAFT=0 DIRECT=1 PIPE=1 COLI_CUDA_PIPE=2.

Controlled three-run comparison on dev

profile	prefill median	decode median	tok/s median	hit	RSS
cap 8, RAM_GB=52	20.99 s	36.54 s	0.88	38.2%	43.10 GB
cap 10, RAM_GB=56	19.12 s	35.15 s	0.91	41.0%	46.03 GB
cap 13, RAM_GB=60	25.24 s	40.87 s	0.78	45.1%	34.33 GB

Individual cap-10 runs were 0.91 / 0.91 / 0.88 tok/s. Relative to the dev cap-8 median, cap 10 improved decode time by 3.8%, throughput by about 3.4%, prefill by 8.9%, and hit rate by 2.8 points.

Cap sweep probes

resolved cap	RAM_GB	prefill	decode	tok/s	hit
8 (3-run median)	52	20.99 s	36.54 s	0.88	38.2%
10 (3-run median)	56	19.12 s	35.15 s	0.91	41.0%
11 (one probe)	58	20.95 s	36.72 s	0.87	42.4%
12 (one probe)	59	21.38 s	37.39 s	0.86	43.8%
13 (3-run median)	60	25.24 s	40.87 s	0.78	45.1%

For cap 13 I requested cap 32 and let the resource guard clamp it safely:

[RAM_GB=60.0] resident 31.4 GB + reserve 7.3 GB
experts 21.5 MB avg x 76 layers -> cap lowered 32->13
(projected peak 59.9 GB)

The hit rate rises monotonically, but performance peaks at cap 10 on this workload. Past that point, the additional cache does not repay its cost.

Corrected CUDA MTP experiment

One cap-13 run with COLI_CUDA_MTP=1 DRAFT=1:

prefill	decode	tok/s	hit	tokens/forward	acceptance
23.76 s	44.22 s	0.72	37.0%	2.00	100% (16/16)

This is a useful negative result: even at perfect acceptance it is slower than DRAFT=0. Aggregate decode expert-disk service rose to 192.347 s and hit rate fell from 45.1% to 37.0%, so the widened expert union outweighed the saved forwards.

Rebuilt current-dev iobench after #1127

Standard 19 MB x 64, 8 threads: buffered 14.07 GB/s, O_DIRECT 11.41 GB/s.

O_DIRECT, 19 MB x 128:

threads	1	2	4	8	12	16
GB/s	10.81	11.88	11.74	12.07	12.50	12.46

The repaired benchmark now shows real but modest scaling, topping out around 12 threads. The original cross-thread comparison should be disregarded; the old 8-thread O_DIRECT number was directionally close.

Bottom line: cap 10 / RAM_GB=56 is the best measured setting; larger caps regress despite higher hit rate, and CUDA MTP depth 1 is negative here even with 100% acceptance. Current-dev baseline decode is effectively flat versus the original published median (0.88 vs 0.89 tok/s).

[Appie-NL]

Follow-up on the same RTX 5090 / 64 GB machine, inspired by the partial/full-mirror results in #1183. This box adds a useful asymmetric case: the secondary NVMe is much slower than the primary, so I chose mirror coverage to target a bandwidth-weighted split instead of copying the entire model and forcing a 50/50 split.

Same measured profile

• dev commit 5c822f71683c2744e115298d8fa2e4aa9cf74d37, native Windows CUDA/sm_120
• Ryzen 7 9850X3D, RTX 5090 32 GB; 478 resident CUDA experts / 10.15 GB
• 64 GB Kingston Fury Beast KF560C30BBAK2-64, XMP DDR5-6000 CL30
• GLM-5.2 int4, cap 10, RAM_GB=56, PIN_GB=20, DRAFT=0, CUDA dense/attention/pipe enabled

The asymmetric storage setup

• primary: WD_BLACK SN8100 2 TB, PCIe 5.0, short CPU-side PCI path
• mirror: Kingston KC3000 2 TB, PCIe 4.0, behind the chipset/switch path

Colibri iobench, O_DIRECT, 19 MiB blocks, 128 blocks:

threads	SN8100	KC3000
1	10.89 GB/s	5.58 GB/s
2	14.66 GB/s	6.34 GB/s
4	14.60 GB/s	6.33 GB/s
8	13.87 GB/s	6.32 GB/s

The plateau ratio predicts a KC3000 share of 6.32 / (14.60 + 6.32) = 30.2%. I therefore did not make a full mirror. mirror_plan.py --budget-gb 220 --reserve-gb 100 selected 79 high-usage shards: 219.2 GiB actual, about 60.6% usage-weighted coverage. Every staged shard passed the planner's size/SHA-256 verification.

Runtime telemetry landed very close to the prediction in every run:

MIRROR: primary 795.30 GB (175384 reads) | mirror1 302.46 GB (61026 reads) — 28% of expert bytes from the mirrors

Controlled A/B

Same fixed prompt as the earlier measurements in this issue, deterministic sampling and settings throughout. I froze the existing routing history with USAGE_SAVE=0 and interleaved the order as A-B / B-A / A-B.

run	single SN8100	partial mirror
1	39.26 s / 0.81 tok/s	36.15 s / 0.89 tok/s
2	36.79 s / 0.87 tok/s	33.12 s / 0.97 tok/s
3	36.69 s / 0.87 tok/s	32.69 s / 0.98 tok/s
median (exact time)	36.79 s / 0.870 tok/s	33.12 s / 0.966 tok/s

That is +11.1% median throughput / -10.0% decode time. Median prefill improved 19.98 -> 18.28 s (-8.5%), median disk service 113.274 -> 93.753 s (-17.2%), and median disk wait 21.301 -> 17.560 s (-17.6%). Hit rate was 41.0% in both conditions.

The longer 128-token check agrees:

	single SN8100	partial mirror	change
decode	146.86 s / 0.87 tok/s	129.74 s / 0.99 tok/s	+13.2% exact throughput
expert-disk service	445.957 s	362.728 s	-18.7%
expert-disk wait	83.446 s	68.096 s	-18.4%
expert-matmul	49.339 s	49.031 s	-0.6%
hit rate	41.1%	41.2%	unchanged

So the second drive helps even though it is less than half as fast, provided mirror coverage is chosen to produce a bandwidth-weighted read split rather than a 50/50 split. The observed 28% mirror traffic versus the 30.2% target is the part I find most useful: it suggests the existing usage-weighted partial-mirror mechanism can also act as a practical load-balancer for heterogeneous drives.

This is only one selected mirror point, not proof that 219.2 GiB is the global optimum. But it gives a concrete reason for a future planner option such as target replica share / measured drive bandwidth: on asymmetric drives, "more mirrored" can eventually oversubscribe the slower device even though the partial mirror is a clear win.

[JustVugg]

Thank you. A 5090 on native Windows with the full CUDA config was missing from the table, and your configuration block is reproducible as written, which is rarer than it should be.

Recorded in docs/benchmarks.md (#1269), so the number is findable by the next person with your hardware instead of only by scrolling the tracker. Closing the issue, not the datapoint.

If you rerun on a later version and the number moves, please open a new one. A datapoint that changed is more interesting than one that held.