feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend

[kreuzzelg]

Depends on: #712 (qwen36 engine) — this branch includes those commits; only the tier commit is new here.

Summary

Applies colibri's placement concept — route → place → overlap → learn — to
Qwen3.6 one level up from the GLM disk tier: RAM → VRAM. All experts stay in RAM; routing heat promotes the hot ones into DEVICE_LOCAL VRAM across one or more GPUs, computed through the existing backend_cuda.cu
expert-group API (no new backend, unlike the Vulkan approach in #602).

• one home device per expert (eid % n_gpus), no duplicates
• tier.h LFRU semantics + hysteresis; parallel warmstart fills the budget before token 1; HEAT_FILE persists heat across runs ("the more you run, the hotter the right experts get")
• async expert groups on all devices; VRAM misses fall back to the CPU int8 path, overlapped with the in-flight groups (as is the shared expert)
• warmstart frees the RAM int8 copies of VRAM residents (rematerialized from packed int4 on LFRU eviction) → 35B int4 fits in ~29 GB RSS
• default build stays CPU-only (inline stubs); CUDA=1 compiles the tier

Try it

Pre-converted self-contained int4 container (all numbers above were measured
with it): https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4

Limitations

• int4 containers only (int8 containers keep the CPU path)
• requires cap == n_experts (full RAM residency)
• single-stream decode (no batching yet)

Thanks on Claude Code

[kreuzzelg]

Rebased onto dev with #762 merged — the async grouped-int4 dispatch this tier rides on — and onto the current #712 branch (serve + gs64), extended for gs64 containers: qt_init takes expert_gs, experts upload as fmt=4 with grouped scales via coli_cuda_tensor_upload_g, CPU-miss fallbacks dispatch through the same grouped GEMV the engine uses.

Verified on a single RTX 3070: output bit-identical to the CPU run on the gs64 container (greedy, 200 tokens), 11.94 tok/s with 3731/10240 experts resident (6.15 GB VRAM), and the serve path rides the tier unchanged (tier init runs before serve_loop).

Staying draft until #712 lands; after that this is one commit on top of it.

[JustVugg]

Same situation as #712 — see my note there. #165 has landed on dev; your render_chat_qwen and the new render_chat_v4 collide textually in c/openai_server.py without contesting anything, but the hunks interleave both function bodies, so the resolution belongs with you rather than with me.

Your CI had never run either — it was stuck in action_required. It is running now, so check the results before you rebase.

[kreuzzelg]

Rebased onto the #712 merge and out of draft — head 34deee3, one commit, 22/22 green. This is the first green run this PR has ever had: it was conflicted continuously, and conflicted PRs get no pull_request runs at all, so CI had never actually seen it.

Thanks for merging #712. Three things followed from that merge, plus one I did not expect.

1. The Makefile rule — neither side was right on its own

#712 moved qwen36 to NOCUDA_CFLAGS, since that file contained no CUDA. This PR adds CUDA to it. Taking my side of the conflict would have broken the tier build; taking the old #713 side would have brought the toolkit dependency back for plain CPU builds, i.e. exactly the thing #712 fixed.

So the flags now follow the same switch the sources already follow:

ifeq ($(CUDA),1)
QWEN36_TIER_SRC = qwen36_tier.c
QWEN36_CFLAGS   = $(CFLAGS)
QWEN36_LDFLAGS  = $(LDFLAGS)
else
QWEN36_TIER_SRC =
QWEN36_CFLAGS   = $(NOCUDA_CFLAGS)
QWEN36_LDFLAGS  = $(NOCUDA_LDFLAGS)
endif

make qwen36 stays toolkit-free; make qwen36 CUDA=1 gets the real thing.

2. g4/u4/d4 are back — and the comment on them was wrong all along

I removed these packed int4 buffers in #712 because nothing there read them. This PR reads them in qt_note() and stage(), so they return. I brought them back as a reverse patch of 67c7490 rather than retyping them, so the restored code is byte-for-byte what was removed.

That made an older defect visible. The comment claimed the buffers were allocated "only when a GPU int4 backend is actually active" — but the code allocated them unconditionally. The +50% expert-cache RSS that #712 removed would have come straight back with this PR, on a build that may not even have the tier compiled in. The allocation is now gated on qt_ready(), which under CUDA=0 is an inline return 0, so a CPU-only build pays nothing. The warmstart's free(e->g) was already guarded on e->g4, so the two stay consistent.

Restored code is not reviewed code. Worth reading the comment against the behaviour, not just diffing the text.

3. Builds

make qwen36	159,032 B, no CUDA symbols
make qwen36 CUDA=1	1,044,120 B, qt_init + cudaMalloc present

What is not verified here

I did not run the tier against the real 35B container in this round — the machine's baseline service is back up and the two must not run concurrently. The end-to-end numbers stand from the earlier runs (bit-identical to CPU at 11.94 tok/s on the 3070, comment 5171528999), and @sehHeiden independently built and ran this branch in #1040. What is new and verified today is that it rebases clean, builds both ways, and passes the full matrix including the ARM job your #1083 added.

Container for reproducing: qwen3.6-colibri int4-gs64, same source as linked in #712.

[kreuzzelg]

Follow-up to my previous comment: the machine freed up, so I ran the two things I had listed as not verified. Both are now measured, on head 34deee3 against the real Qwen3.6-35B-A3B int4-gs64 container (22 GB), 49-token prompt, 200 generated tokens, OMP_NUM_THREADS=12 OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close.

1. Tier vs CPU parity on the full model

	CPU only	CUDA tier, 2 GPUs
Speed	1.44 tok/s	10.05 tok/s (7.0×)
Peak RSS	24.66 GB	29.73 GB
Expert cache hit rate	91.6 %	90.6 %

Output is bit-identical (cmp on the full 200-token generation, not a spot check). Hardware: Quadro RTX 4000 + RTX 3070, 8 GB each. The tier run was cold — no heat table existed yet, so this is the pessimistic figure.

2. The qt_ready() gate, measured rather than argued

In my last comment I claimed the unconditional allocation of g4/u4/d4 would have brought back the RSS regression that #712 removed. I did not want to leave that as an assertion, so I built the same source twice — the only difference being if (qt_ready()) vs if (1) — and ran both CPU-only on the real container:

CPU-only run, identical prompt	Peak RSS
without the gate (old behaviour)	26.19 GB
with the gate	18.86 GB

7.33 GB, a 39 % surcharge for buffers a CPU-only build never reads. Output identical, speed identical (0.32 vs 0.33 tok/s, noise). I verified the two binaries actually differ before trusting the result — with the gate folding to a constant under CUDA=0 it was worth checking that the compiler had not made the experiment vacuous.

Also green locally

tests/test_qwen36_ctx (14 checks), tests/test_makefile_cuda_scope.py (5), tests/test_family_registry.py (12) — alongside the 22/22 in CI.

Still not claimed

Long-context behaviour of the tier is unchanged from the earlier measurements and is not re-measured here; the second GPU's advantage shrinks with context depth (1.20× at 4k → 1.03× at 128k in my fork experiments), because the CPU-side KV scan dominates. That is a property of the engine, not of this PR, and I am not proposing anything about it here.

[JustVugg]

Running the two unverified items after merge, unprompted, with a cmp on the full generation and a compiler-vacuity check on your own experiment — this is the follow-through that makes merging on trust cheap. 1.44 → 10.05 tok/s (7.0×) on two 8 GB cards, bit-identical, cold-start is the headline of the upcoming release notes, and the 7.33 GB the qt_ready() gate saves CPU-only builds goes right next to it. The long-context caveat is noted where it belongs: engine property, tracked, not this PR's problem.