[Performance]: GLM-5.2 (744B MoE) on a single AMD R9700 32 GB: 0.15 tok/s, 80% expert hit — capacity-bound benchmark

[limcheekin]

Commit

81f08a0

Hardware and storage

Hi there,

I use Opus 4.8 to drive the benchmark, it add the following benchmark results to the docs/benchmarks.md, appreciate if you have any idea to improve the performance:

Community benchmarks (measured)

Real numbers from real machines, stock build (setup.sh, gcc 13), greedy decoding, --ngen 32, MTP active:

machine	disk (iobench, 19 MB × 64, 8 threads)	config	measured
Intel i5-8600K (6C/6T, AVX2) · native Linux · 125 GB RAM · SATA SSD · AMD Radeon AI PRO R9700 32 GB (gfx1201/RDNA4, ROCm 7.2.2 HIP backend), measured 2026-07-24	— (SATA SSD, not iobench'd)	HIP build · CUDA_EXPERT_GB=30 CUDA_RELEASE_HOST=1 DIRECT=0 · warm cache · 1586 pinned experts (30 GB VRAM)	0.15 tok/s · hit 80.2% · RSS 97.7 GB (64-tok, MTP acc 0%) · at 26 GB budget → 0.12 tok/s / 77%

AMD single-card setup (ROCm/HIP, R9700) — measured 2026-07-24

First end-to-end AMD-discrete datapoint (the GPU backends matrix
had a hip-test pass on gfx1201 but no tok/s). Two setup notes not in the build docs:

# 1) Build the HIP backend. The documented command links as a PIE and fails
#    (backend_cuda.o is not -fPIC), so force position-independent code:
rm -f c/backend_cuda.o c/colibri c/glm
ROCM_HOME=/opt/rocm-7.2.2 make -C c glm HIP=1 HIP_ARCH=gfx1201 \
  HIPCCFLAGS="-O3 -std=c++17 -x hip --offload-arch=gfx1201 -fPIC -Wall -Wextra"

# 2) The `coli` wrapper is NVIDIA-only: cuda_binary() greps ldd for libcudart
#    (a HIP build links libamdhip64), and resource_plan.discover_gpus() only calls
#    nvidia-smi. So `coli run --gpu` / doctor report "no GPU". Drive the engine
#    directly — it enumerates the AMD device via HIP itself:
SNAP=/path/to/glm52_i4 NGEN=64 \
  PROMPT='[gMASK]<sop><|user|>...<|assistant|><think></think>' \
  COLI_CUDA=1 COLI_GPUS=0 HIP_VISIBLE_DEVICES=0 \
  CUDA_EXPERT_GB=30 CUDA_RELEASE_HOST=1 DIRECT=0 ./c/colibri 8

Result: HIP init clean on gfx1201/ROCm 7.2.2, coherent greedy output. Raising the
VRAM expert budget 26 → 30 GB (1359 → 1586 resident experts) lifted decode
0.12 → 0.15 tok/s and hit rate 77 → 80%; a warm-cache priming pass cut
prefill 188 → 137 s (buffered pread, DIRECT=0) but barely moved steady-state
decode. Hit rate plateaus at ~80% even at 30 GB: this prompt's working set
(~2757 hot experts) exceeds what one 32 GB card holds, so ~20% of routes stay cold
disk reads. It is capacity-bound, not compute-bound (decode wall ≈ 280 s
disk-wait + 160 s compute over 64 tokens); the compute-only floor is ~0.4 tok/s.
More residency needs more VRAM (more cards), not tuning.

Software environment

OS Ubuntu 24.04.4 LTS · kernel 6.8.0-134-generic (x86-64)
CPU Intel Core i5-8600K (6C/6T, AVX2) @ 3.60 GHz
RAM 125 GiB
Disk 3.6 TB SATA SSD (/dev/sda1), model on /mnt/data (2.3 TB free)
GPU AMD Radeon AI PRO R9700, 32 GB VRAM (34.2 GB reported) · gfx1201 (RDNA4)
ROCm / HIP ROCm 7.2.2 · HIP 7.2.53211 · AMD clang 22.0.0git (roc-7.2.2)
Host compiler gcc 13.3.0 (Ubuntu)
Engine colibri 1.1.1 (c/colibri), built with the HIP backend
Model GLM-5.2 int4 + int8 MTP — /mnt/data/models/GLM-5.2-colibri-int4-with-int8-mtp
(~384 GB on disk, 144 shards, 9.9 GB dense resident, 19,456 experts)

Build and benchmark commands

# --- 1. Build the colibri HIP engine for gfx1201 -----------------------------
# NOTE: the documented `make ... HIP=1` links as a PIE and fails
# (backend_cuda.o is not position-independent), so add -fPIC explicitly.
cd /mnt/data/colibri
rm -f c/backend_cuda.o c/colibri c/glm
ROCM_HOME=/opt/rocm-7.2.2 make -C c glm HIP=1 HIP_ARCH=gfx1201 \
  HIPCCFLAGS="-O3 -std=c++17 -x hip --offload-arch=gfx1201 -fPIC -Wall -Wextra"
ldd c/colibri | grep amdhip     # verify libamdhip64 is linked

# --- 2. Run the benchmark (max VRAM budget + warm page cache) ----------------
# The `coli` wrapper is NVIDIA-only (cuda_binary() greps ldd for libcudart;
# resource_plan.discover_gpus() only calls nvidia-smi), so it refuses --gpu on a
# HIP build. Drive the engine binary directly — it enumerates the AMD GPU via HIP.
# argv[1] = prompt cap; everything else is read from the environment.
export SNAP=/mnt/data/models/GLM-5.2-colibri-int4-with-int8-mtp
export PROMPT='[gMASK]<sop><|user|>Explain what a mixture-of-experts model is in two sentences.<|assistant|><think></think>'
export COLI_CUDA=1 COLI_GPUS=0 HIP_VISIBLE_DEVICES=0   # select the AMD device
export CUDA_EXPERT_GB=30       # max expert tier on a 32 GB card (2 GB reserve)
export CUDA_RELEASE_HOST=1     # drop RAM copies of VRAM experts -> more warm RAM
export DIRECT=0                # buffered pread -> OS page cache is used (warmable)
export COLI_POLICY=quality

# prime pass (warms page cache + expert placement / .coli_usage), then measure:
NGEN=32 ./c/colibri 8          # prime
NGEN=64 ./c/colibri 8          # measured (read the steady-state tail [t=N ... tok/s])

Results

# --- Result -------------------------------------------------------------------
# CUDA_EXPERT_GB=30, warm: 0.15 tok/s decode · 80.2% expert hit · 1586 experts
#   resident in 30.00 GB VRAM · peak RSS 97.7 GB · prefill 137 s (warmed from 188 s).
# Baseline CUDA_EXPERT_GB=26: 0.12 tok/s · 76.9% hit · 1359 experts (25.99 GB).
# Capacity-bound at ~80% hit on one 32 GB card (working set ~2757 hot experts);
# compute-only floor ~0.4 tok/s. More throughput needs more VRAM (more GPUs).

Baseline comparison

No response

[noobdev-ph]

Try using vuklan first. You can refer to this discussion - #523

[JustVugg]

This is a genuinely excellent datapoint — the first AMD-discrete tok/s on the tracker (RDNA4/gfx1201), and your capacity-bound diagnosis is exactly right. Confirming and adding to it:

Your diagnosis is correct. 80% hit plateau at 30 GB + decode wall ≈ 280 s disk-wait vs 160 s compute over 64 tokens = capacity-bound, not compute-bound. The ~2757-expert working set doesn't fit one 32 GB card, so ~20% of routes are cold SATA reads. "More residency needs more VRAM, not tuning" is the honest conclusion. The compute-only floor you derived (~0.4 tok/s) is the ceiling a second card would approach.

Two real tooling bugs you hit — both ours, both worth fixing:

1. make ... HIP=1 links as PIE and fails because backend_cuda.o isn't built -fPIC. Your -fPIC workaround is the fix; the Makefile should pass it for HIP builds so the documented command just works.
2. The coli wrapper is NVIDIA-only — cuda_binary() greps ldd for libcudart and discover_gpus() only calls nvidia-smi, so coli run --gpu / coli doctor report "no GPU" on a HIP build and you had to drive ./c/colibri directly. Detection should recognise libamdhip64 + rocm-smi.

I'll put a PR up on dev for both so the next AMD user doesn't reverse-engineer the invocation. Thank you for documenting it precisely enough to fix.

One lever you're leaving on the table: MTP acc 0%. On a disk-bound decode, accepted MTP tokens are worth the most — a verified draft amortizes the same expert working-set load across two tokens, so at your 280 s-disk / 64-tok ratio, non-zero MTP acceptance would move throughput more than any VRAM tuning short of a second card. 0% acceptance on a -with-int8-mtp container is unexpected (the int8 head exists precisely so MTP works, see #8). Could you run a quick check — does the log show [MTP] active and draft=N (N>0)? If MTP is active but every draft is rejected on the HIP path, that's a separate GPU-MTP bug I'd like to chase (it may be that MTP verification isn't wired on the HIP backend yet). A CPU-only run of the same prompt (COLI_CUDA=0, small --ngen) that shows non-zero acc would confirm it's HIP-path-specific.

For the 20% cold reads: a second SSD via COLI_MODEL_MIRROR (#421) or COLI_MODEL_DIRS (#469) raises cold-read bandwidth directly; NVMe over SATA is the bigger single win. But you're right that the structural fix is VRAM.

Would you like me to add this row to docs/benchmarks.md myself, or are you opening a PR with it? Either works — it's a valuable measurement to keep.

[RDouglasSharp]

I filed PR #585 because I think it gates having realistic numbers for your request on METAL. I plan to work on implementing METAL-based fmt:4 attention, at which point I can send you a PROF=1 breakdown of timing. Do you have a specific preferred task that you run to obtain your other timings?

[JustVugg]

Closing this as recorded, not as ignored — and the distinction deserves stating, because closing someone's contribution without one reads as dismissal.

Hardware datapoints are contributions of data, not requests for work. They stay valuable permanently: someone with your hardware, searching before they commit to a 400 GB download, wants exactly this. But sitting open in the issue tracker they read as unfinished tasks, and fifteen of them made the project look like it had sixty open problems when it had about twenty. That is a signal-to-noise cost paid by everyone who arrives wanting to help and cannot find where.

So: the numbers stay, the "open task" framing goes.

• The issue and its whole thread remain readable and searchable — nothing is deleted.
• Discussion #208 is the standing home for benchmark datapoints, and where new ones are most useful.
• docs/benchmarks.md is where datapoints that map a regime boundary get written up.

Reopen it without asking if there is a defect or an open question in here that I read past. Several benchmark reports in this tracker turned out to contain real bugs — #718's SMT collapse and #634's two defects are staying open for exactly that reason — and if yours is one of them I would rather be corrected than have it quietly closed.

Thank you for taking the time to measure and report it properly.