feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE

[kreuzzelg]

Summary

The idea of this and next PR is to optimize loading experters (for MoE model) from RAM to VRAM when the VRAM is limited, so that for example the "Qwen3.6:35b" model can run at a single GPU with 8 GB VRAM with acceptable performance.

Measured (Threadripper 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)

configuration	decode tok/s	VRAM hit rate	peak RSS
CPU only	~2.5	–	~13 GB
1× 8 GB GPU	9.9	95 %	40 GB
2× 8 GB GPUs	11.3	100 %	29 GB
Ollama 2x 8GB GPUs	10.5	–	~40 GB

Relationship to #602: the engine core (hybrid forward, container format, converter, tiny/oracle validation) originates from @minne100's PR; this PR retargets it to dev, adds substantial CPU performance work and converter fixes, and drops the Vulkan/serve parts per review guidance.

Self-contained engine for Qwen3.6-35B-A3B (35B total / 3B active, 256 experts/layer, Apache-2.0). This is the "engine first" slice requested in the #602 review (thanks for the guidance!): no shared GLM/Inkling files are touched, no GPU backend is included — those come as separate follow-ups.

What's included

• c/qwen36.c — 40-layer hybrid forward: 10x Gated Attention (GQA, partial RoPE rope_dim=64, per-head q/k RMSNorm, output gate) + 30x Gated DeltaNet (causal depthwise conv ring + recurrent gated delta rule), streaming MoE with per-layer LRU expert cache + shared expert, PILOT router-lookahead prefetch, COLIBRI_RESIDENT pinning, per-phase timers (COLI_TIMERS=1).
• CPU performance: optional per-row int8 quantization of the large dense matrices (COLI_DENSE_I8=1 default; f32 originals freed; =0 restores the bit-exact f32 reference path), hand-written AVX2/FMA int8 GEMV, DeltaNet recurrence parallelized per value head. ~7x vs the scalar baseline.
• c/tools/convert_qwen36.py — HF→container converter (true int4, per-row scales, --selftest). Containers now run out of the box: flat config.json (VL checkpoints nest dims under text_config) and bundled tokenizer.json.
• c/tools/make_qwen36_tiny.py, make_qwen36_oracle.py — weight-free validation without the 70 GB checkpoint.
• docs/qwen36-phase01.md, qwen36-phase02.md.

Validation

• converter --selftest PASS (int4 pack/unpack exact)
• tiny-model logits bit-identical to the validated reference (cosine 1.0)
• real 35B int4 container: coherent generation; dense-int8 logits cosine
  0.9992 vs f32 reference (COLI_DENSE_I8=0 → bit-exact)

Try it

A pre-converted, self-contained int4 container (tokenizer + flat config bundled, produced with this PR's converter) is available: https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4

Notes

• The earlier published i4/i8 HF containers predate the converter fixes: they lack tokenizer.json and ship the nested VL config (workaround: TOK= and a flattened config.json; new conversions are self-contained).

Thanks on Claude Code.

[jazir555]

Can you add Qwen 27B support? It would be awesome to use since it's pretty much entirely out of reach for my system right now. 32 GB ram, 12 GB 4070 super, 12700k. 27B support with your method would make it usable and fit entirely in VRAM i think

[minne100]

Hi jazir555, Thanks for the suggestion! Unfortunately I can't add dense-27B support to colibri the way you're hoping, and here's the honest reason why. colibri's whole trick is expert streaming ― for MoE models we only keep the active experts (plus shared experts + attention) resident and stream the rest from disk on demand. That's what lets a 30B/3B-active model run on a 16 GB laptop. Your 12 GB 4070 Super is exactly the kind of machine this is built for if the model is MoE. A dense 27B (e.g. Qwen3-32B / Qwen2.5-32B) has no experts to stream ― every parameter is needed for every token. Even in int4 that's ~16 GB, which already exceeds your 12 GB VRAM before you add KV cache and CUDA overhead. The only way to run it would be CPU offload + paging/swap, which is slow and is precisely what llama.cpp / KTransformers already do very well. Building dense-27B support into colibri would just be a worse llama.cpp, so it's not worth doing. If you actually meant Qwen3-30B-A3B (the MoE: 30.5B total / 3.3B active), that's a completely different story ― it fits your 12 GB card comfortably (active weights are only ~1.65 GB in int4, and pinning the hot experts keeps the whole working set in VRAM). I'd happily add that. Just say the word. Cheers

…

________________________________ [https://avatars.githubusercontent.com/u/31750063?s=20&v=4]jazir555 left a comment (JustVugg/colibri#712)<#712 (comment)> Can you add Qwen 27B support? It would be awesome to use since it's pretty much entirely out of reach for my system right now. 32 GB ram, 12 GB 4070 super, 12700k. 27B support with your method would make it usable and fit entirely in VRAM i think ― Reply to this email directly, view it on GitHub<#712?email_source=notifications&email_token=AAX3ZWIUIO6UAHNLEONKX3L5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5147442249>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAX3ZWJILVQV3ASEJ542MO35HUCVJAVCNFSNUABGKJSXA33TNF2G64TZHMYTEOBWGA4DAMZZG45US43TOVSTWNJQGE3TMOJXGMYTPILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AAX3ZWNFKWKIGW236QZGRJD5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/AAX3ZWOUE3MO3VRVY7KI47T5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[LionelColaso]

@minne100 already exists #544
@kreuzzelg support for KAT-Coder-V2.5-Dev would be awesome

[JustVugg]

Reviewed this properly, and I owe you both a compliment and a correction of something I said internally about it.

The file shape is exactly right. c/qwen36.c as a single engine, a converter, two oracle-fixture generators, six lines of Makefile, and zero changes to any existing file. That is the shape inkling.c and kimi_k3.c landed in and it is what we ask new engines for. You got there without being told, and it made this quick to read.

But it cannot be driven by coli yet, and that is the blocker rather than anything about the engine:

	kimi_k3.c	qwen36.c
emits the READY handshake	✅	❌ none
serve mode (SERVE=1)	✅	❌ none
known to c/coli	✅ (#676)	❌ not touched

So coli chat, coli web and coli serve would all fail: the launcher does not know the arch exists, and even pointed at the binary the engine does not speak the protocol. Users do not run engines directly — they run coli chat. An engine that only generates from argv is, in practice, not reachable.

What it takes

1. c/coli, two lines.

# model_arch(), c/coli:160
if "qwen" in model_type: return "qwen"        # config.json says model_type: qwen3_5_moe

# engine_for(), c/coli:176
name = "inkling" if arch == "inkling" else "kimi_k3"   # -> add "qwen36"

coli reads the model's config.json, picks the binary, and for non-GLM engines spawns the local gateway and attaches the TUI — so chat, web, serve and the OpenAI-compatible API all come from those two lines plus the chat template.

2. Serve mode in the engine. kimi_k3.c's serve_loop is the shortest example: emit \x01\x01READY\x01\x01\n, then a STAT line, then read SUBMIT/CANCEL frames. #676 is the worked example of the whole thing for Kimi.

3. One thing that will bite you on Windows, which we only found today. Call coli_serve_binary_mode() from compat.h before emitting the sentinel. On Windows the CRT rewrites the trailing \n as \r\n, the gateway never matches the sentinel and the session hangs forever with no error. inkling.c and kimi_k3.c both had this bug and it cost a user a 42-minute load followed by silence (#748, fixed in #749). It is one line, and you get it for free by adding it now rather than shipping it and finding out.

Two questions

Which container should people use? Qwen3.6-35B-A3B upstream is bf16, so it needs converting — convert_qwen36.py does that, and you have also published Kreuzzelg/qwen36-35b-a3b-colibri-i4 at ~20 GB. Should the docs point at your prebuilt container as the easy path, with the converter as the do-it-yourself route? That is how docs/inkling.md is structured and it works well.

Per-row int4 or gs64? Your container is per-row. On GLM, per-row int4 was the root cause of #455's think-mode loops and never-terminating generations, and the group-scaled gs64 container fixed those controlled A/Bs — that is why the GLM README steers people away from the per-row mirrors. With moe_intermediate_size=512 the groups on Qwen are short, so I would expect per-row to lose more here rather than less, but I have not measured it on this model and would not want to assert it. If you can run the same prompt on both containers it would settle the question, and it is the kind of measurement that is much cheaper for you than for us.

Related

#729 (@minne100) is a Vulkan MoE GEMV backend for integrated/AMD GPUs, explicitly written as the GPU half for this engine and currently a draft blocked on this PR landing. Worth the two of you talking — there is also a question there about a vendored 549 KB vulkan_core.h versus the system header the merged Vulkan backend (#418) uses, which is easier to settle between you than after the fact.

No rush from us, and none of this is a criticism of the engine — 2,009 lines that build clean and leave every existing file untouched is a good PR. It just needs the front end before people can actually reach it.

[minne100]

@maintainer — pushed the coli integration + SERVE=1 gateway protocol and addressing the review point by point. (Note: I can't edit this PR's head/body via API — I have no push access to the kreuzzelg/colibri head branch and I'm not a collaborator on upstream — so here's everything as a comment. The updated branch is pushed to minne100/colibri:feat/qwen36-gpu-residency-serve, now at bcae842, sitting on top of the residency/GPU fixes. Happy to retarget this PR's head to that branch or open a fresh PR from my fork — see last point.)

1. coli reachability — DONE

c/coli now recognizes the engine:

• model_arch(): if "qwen" in model_type: return "qwen" (before the glm fallback).
• engine_for() / need_model() map qwen → qwen36 (binary + build target).
• chat banner shows model_id="qwen36-colibri" for the qwen arch.
  So coli <qwen36-model> now drives qwen36.exe exactly like it drives glm / inkling / kimi. Commit bcae842.

2. SERVE=1 gateway protocol — DONE

Added a serve mode to c/qwen36.c that speaks the existing non-GLM wire protocol (the one kimi_k3.c / inkling.c already use):

• READY sentinel + STAT <rss> + SUBMIT <id> <slot> <plen> <max_tok> <temp> <top_p> + ACCEPT + DATA <id> <n> + DONE <id> STAT <gen> <tps> <hit%> <rss> <np> <limited> (+ PROF). No END frame — that's GLM/colibri.c-only. CANCEL <id> is handled by the gateway.
• Sampler: ported kimi's top-p / qsort sampler. Previously generate() was greedy-only; serve_sample() now honors temp / top_p.
• EOS: Qwen3 <|im_end|> = 151645 (Q36_EOS env override).
• Windows fix: inlined _setmode(_fileno(stdout/stdin), _O_BINARY) inside serve_loop(). coli_serve_binary_mode() from fix(win): put stdout in binary mode before the serve handshake (closes #748) #749 is not in base yet (confirmed absent), so I inlined it with a // TODO: switch to shared helper once #749 lands. Will swap it out the moment fix(win): put stdout in binary mode before the serve handshake (closes #748) #749 merges.

3. Which container to document

Plan: document the prebuilt container plus the self-convert path, following the docs/inkling.md pattern.

• Namespace correction: the prebuilt containers are published under the minne100/ Hugging Face namespace — minne100/qwen36-35b-a3b-colibri-i4 (and -i8) — not kreuzzelg/. The GitHub fork account (kreuzzelg) and the HF publish account (minne100) are different handles, so the container path shouldn't be derived from the fork name.
• Docs will point at the prebuilt i4 by default and at c/tools/convert_qwen36.py (true-int4 by default) for self-converting from the upstream HF safetensors.

4. per-row int4 vs gs64

Recommendation: switch to group-scaled int4 (gs64, group size 64) as the default converter output, replacing the current per-row int4.

• Rationale: GLM issue GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe #455 showed per-row int4 triggered think-mode infinite loops; gs64 fixed it. For Qwen, moe_intermediate_size=512 makes the rows short, so per-row quantization loses proportionally more.
• Caveat (honest): I have not yet run an A/B on Qwen to measure the actual accuracy delta. Before flipping the default I'll (a) add gs64 to convert_qwen36.py, (b) produce a gs64 container, and (c) A/B it against per-row int4 (logit-cosine vs the HF oracle + a generation-quality check). Current default stays per-row int4 until that's measured — I won't change the default on assertion alone.

5. #729 / Vulkan header coordination

qwen36.c currently vendors vulkan_core.h (~9.8k lines) to support the in-shader int4 decode path. You flagged this against #418's system-header approach — agreed it's the wrong end state.

• Plan: align with Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) #418. Once Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) #418 decides the header strategy (system vulkan header vs one shared vendored copy), I'll reconcile qwen36's vulkan_core.h to that single source so we don't ship two divergent copies. I'll hold the vendored copy as-is for now and switch to the agreed header when Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) #418 lands.

6. Push / head — action needed from you

Because I can't push to the kreuzzelg/colibri head branch that this PR tracks (and can't edit the PR head/body via API), the updated code lives at minne100/colibri:feat/qwen36-gpu-residency-serve (bcae842). Please pick one:

• (a) retarget this PR's head to minne100:feat/qwen36-gpu-residency-serve, or
• (b) I open a fresh PR from my fork against dev.

Either way the coli + SERVE work above is already in that branch. Let me know which you prefer and I'll proceed (and fold in the gs64 converter + #729 header reconcile once measured/decided).

[kreuzzelg]

@JustVugg — ran the per-row vs gs64 A/B you asked for. Short version: gs64 is measurably more accurate on Qwen (≈44 % less quantization error), but per-row does not reproduce the #455 loop symptom here — so switching is a quality improvement, not a bug fix. Details below; both containers built from the same bf16 base, same engine, same prompts, only the expert quant differs.

1. Accuracy — first-token logit distribution vs an int8 anchor

int8 (per-row) container as the near-lossless reference; both int4 containers compared against it. 4 prompts (MoE-explain, a multi-step math word problem, a primes/twin-primes list, a self-critique task).

prompt	cosine per-row	cosine gs64	KL per-row	KL gs64
p1	0.99141	0.99568	0.0673	0.0620
p2	0.98053	0.99237	0.1439	0.0446
p3	0.99061	0.99531	0.0783	0.0363
p4	0.98855	0.98915	0.1470	0.1751
mean	0.98777	0.99313	0.1091	0.0795

gs64 wins on cosine on all 4 prompts and on KL on 3 of 4 (p4 is a wash on cosine and slightly worse on KL — honest outlier, not cherry-picked out). Mean cosine error to the anchor drops 0.01223 → 0.00687, i.e. ~44 % less. So your intuition that per-row loses more with moe_intermediate_size=512 holds — just not dramatically.

2. Behavior — the #455 symptom does not reproduce on Qwen

Same 4 prompts, greedy decode, 512 tokens, both containers. I checked for the degenerate-repetition / non-termination pattern from #455 (top-4-gram repeat count, unique-token ratio):

	per-row	gs64
max 4-gram repeat (any prompt)	2	2
unique-token ratio range	0.53–0.71	0.54–0.73
runaway loops	none	none

Both generate healthy, non-repetitive text on all 4 prompts. Whatever made per-row int4 loop on GLM think-mode, it isn't firing on Qwen3.6-35B-A3B here. (Caveat: this engine's argv text mode generates a fixed token count and doesn't stop on EOS, so this measures degeneration, not natural termination — the serve path with EOS handling would be the place to confirm termination.)

3. Recommendation

Make gs64 the default converter output — it's a free accuracy win (44 % lower logit error, no measured downside except ~10 % larger container: 20 → 22 GB for the 35B). But since per-row is not broken on Qwen, I'd suggest keeping per-row selectable (--gs 0) rather than removing it, and not treating existing per-row containers as unsafe.

Implementation is already on the branch (gs64-ab): convert_qwen36.py --gs N (group-scaled expert quant, expert_gs in meta) + engine support (matmul_q_gs scalar + AVX2, sized from expert_gs), validated on the tiny model (gs beats per-row vs the int8 reference) and on the real 35B above. Happy to fold it into whichever branch becomes the merge head.

Containers (both self-contained, produced with this converter):

• per-row int4 (current PR default): https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4
• gs64 int4 (this A/B, needs the gs64-ab branch): https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64

Raw numbers + harness are in the branch; happy to paste the full per-prompt
table or the harness script if useful.

Thanks to Claude :-)

[kreuzzelg]

Hi @minne100 — thanks for turning the coli + SERVE=1 integration around so fast, that was the real blocker and you've solved it. Answering your point 6 directly: please open a fresh PR from your fork against dev. A cross-fork head retarget isn't possible on our side anyway, so a new PR from minne100/colibri is the clean path, and it makes sense for the engine PR to be authored by you since the engine is yours.

One thing we'd like to keep from #712 so it isn't lost when this one closes — all small and self-contained, on top of the base engine:

• CPU decode perf: hand-written AVX2/FMA int8 expert GEMV, optional per-row int8 quantization of the dense matrices (COLI_DENSE_I8, f32 freed after), DeltaNet recurrence parallelized per value head. ~7× over the scalar path.
• Converter run-out-of-the-box fixes: bundle tokenizer.json, write a flat config.json for the VL checkpoint (keep the original as config.hf.json).
• gs64 group-scaled expert quant (--gs, expert_gs in meta + engine matmul_q_gs): the A/B above shows ~44 % lower logit error vs per-row.

Whatever's least work for you: we can (a) hand you these as cherry-pickable commits / a patch to fold into your PR, or (b) submit them as one small follow-up PR stacked on yours. We slightly lean toward (b) since the maintainer asked for small reviewable PRs, but your call — you own the engine head.

Once your PR is the engine base, we'll close #712 and rebase our #713 (CUDA VRAM expert tier) onto your branch so it stacks cleanly on top. #713 stays our PR (dedicated-NVIDIA path), your Vulkan #729 stays yours (iGPU/AMD) — complementary, as discussed.

On the containers: no ownership issue from our side — we published the two under the Kreuzzelg/ HF namespace only because that's the GitHub handle here. The converter now makes self-contained containers either way, so point the docs at whichever namespace you and the maintainer prefer (yours is fine); we'll keep ours as mirrors. And agreed on the vulkan_core.h question — best reconciled with #418 before it lands, between the two of you.

Just say which option (a/b) you prefer and I'll get our three items over to you.

---

[kreuzzelg]

@minne100 — to save you the wait on whichever option you pick, I've prepared our three items as clean single-purpose commits on top of your bcae842, so they cherry-pick without conflicts:

Branch: kreuzzelg/colibri:qwen36-improvements-for-minne100

1. fix(qwen36): self-contained containers — bundle tokenizer.json, flat config.json
2. perf(qwen36): AVX2 int8 expert GEMV + dense int8 + parallel DeltaNet
3. feat(qwen36): group-scaled int4 (gs64) — the ~44 % accuracy win from the A/B

Each builds and passes tiny-model validation on your branch (per-row unchanged, grouped path verified, dense-int8 cosine 0.9992 / bit-exact with COLI_DENSE_I8=0).

git remote add kreuzzelg https://github.com/kreuzzelg/colibri
git fetch kreuzzelg
git cherry-pick c1677f6 ca9cca2 94495ec   # or pick a subset

If you'd rather I open them as a follow-up PR on top of your fresh PR (option b), just say so. .patch files are attached in case the SHAs drift.

[kreuzzelg]

Heads-up on a gs64 + GPU-tier gap I hit while benchmarking, so nobody trips over it:

gs64 containers don't run on the CUDA VRAM tier yet — gs64 is CPU-path only right now.

The two pieces were built on separate branches and haven't met:

• gs64 expert quant added a CPU GEMV (matmul_q_gs) + expert_gs in meta; the engine loader sizes the scale buffers from it.
• the CUDA VRAM tier (feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend #713) uploads experts hard-coded as fmt=2 (per-row int4) and its loader expects per-row scales.

So a gs64 container loaded by the tier binary is rejected at load (scale array is 49152 elems — expected 3072), and there's currently no single build that has both. Practical consequence:

• the per-row int4 containers are the ones that run on the GPU tier today (~11 tok/s on 2×8 GB);
• the gs64 containers (the ~44 % more accurate ones) currently only run on the CPU path (~2.5 tok/s).

The good news: backend_cuda.cu already has fmt=4 (grouped int4) kernels, so extending the tier to gs64 is a wiring job — thread expert_gs through qt_note/upload and send fmt=4 with the grouped scales instead of fmt=2. I'm planning to do that so gs64 gets the GPU speed too, and I'll post gs64-on-GPU tok/s once it's in. Flagging now so the published gs64 containers aren't mistaken for GPU-tier-ready.

[kreuzzelg]

Follow-up on the gs64 note above: the reason gs64 containers were CPU-only so far is a backend gap, not an engine limitation — the async CUDA expert-group path (coli_cuda_expert_group_issue) never dispatched grouped
int4 (fmt=4) and silently fell back to a per-row scale. That is now fixed in #762 (standalone against dev, backend + test only — no overlap with this PR or #713).

Once #762 lands, the gs64 containers (Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64) run on the CUDA VRAM tier as
well: verified downstream on the 35B container, CPU vs 2-GPU tier at logits cosine 1.0000000 over 24 greedy tokens, token-identical output.

[rofl0r]

how about cherry-picking bcae842 onto this branch rather than starting over with the original PR that was dead for a week until you revived it here? the same might happen again - it appears the original author has way less time to work on this than you do.

[kreuzzelg]

@LionelColaso — the pre-converted container is up:
https://huggingface.co/Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64 (~22 GB, self-contained, Apache-2.0).

Recap from the verification above: KAT-Coder-V2.5-Dev shares the Qwen3.6-35B-A3B architecture field for field, so the qwen36 engine runs it with zero code changes. I published the gs64 (group-scaled int4) variant — on the sibling
base model it cuts first-token logit error by ~44 % vs per-row, and since the async grouped-int4 backend fix (#762) it runs on the CUDA VRAM expert tier at full speed: 11.3 tok/s warm on 2×8 GB, 10.4 tok/s on a single 8 GB card
(200-token decode, TTFT ~2 s, peak RSS 30/40 GB).

Which commits you need (until the PRs land):

• GPU tier + gs64 (recommended): branch gs64-gpu (de5dde7) — tier fmt=4 upload + upstream-dev backend incl. the fix(cuda): dispatch grouped int4 (fmt=4) in the async expert-group path #762 fix. fix(cuda): dispatch grouped int4 (fmt=4) in the async expert-group path #762 is required for correct gs64 output on the tier (the async decode path otherwise applies per-row scales to grouped containers, silently).
• CPU only: branch gs64-ab (2f9aab2).
• Build/run commands are on the model card.

One caveat worth knowing: there are two KAT-Coder-V2.5 versions. The paper's scores are for the commercial flagship (e.g. SWE-Bench Pro 65.2, Terminal-Bench 2.1 60.7); the open weights — and this container — are KAT-Coder-V2.5-Dev (35B/A3B, text-only, Apache-2.0), which reports considerably lower agentic-coding scores (SWE-Bench Pro 45.96, Terminal-Bench 2.1 41.02, but SWE-Bench Verified 69.40). The model card has the full comparison table, so nobody benchmarks the Dev container against flagship expectations. Also still true: it's a Dev checkpoint — wire up its chat template + EOS for real use; in raw completion mode it keeps writing after the code block.

Since you seem to follow the KAT releases: do you know anything more about the commercial flagship V2.5 — parameter count, architecture, whether it stays closed or open weights are planned? Kwaipilot doesn't disclose specs in the
paper, and it would help to judge how far the open Dev checkpoint is from the scores people usually quote.

[kreuzzelg]

Front-end wiring is in — done the way @rofl0r suggested: @minne100's bcae842 cherry-picked with authorship preserved (coli's model_arch/engine_for/need_model + the SERVE=1 gateway loop), with the inlined Windows fix switched to compat.h's coli_serve_binary_mode() now that #749 is merged, as that commit's own message intended.

Driving the handshake end-to-end then found three more gaps, all fixed on the branch:

1. Serve mode required an argv prompt file. main loaded argv[3] (default ref.json) before the SERVE branch, so an engine launched exactly the way coli launches it — SERVE=1, no prompt argument — exited 1 before ever emitting READY.
2. The EOS id was the 151k-vocab constant. serve_one compared against a hardcoded 151645; Qwen3.6's 248320-token vocab puts the specials at 248044+ (<|endoftext|> 248044, <|im_end|> 248046), so EOS never matched and every turn ran to max_tok. Now resolved from the tokenizer's added_tokens — which is exactly what upstream generation_config.json declares (eos_token_id: [248046, 248044]).
3. The gateway had no qwen template family. openai_server.py would have rendered GLM frames. render_chat_qwen mirrors Qwen3.6's chat_template byte for byte in the text-only subset (inkling's approach). One non-obvious detail the template encodes: the generation prompt must open the <think> block — the model was never trained on a bare assistant\n state, and greedy argmax there is an EOS special (measured: gen=0 on the first token).

Verified: READY/STAT → SUBMIT → ACCEPT → DATA → DONE transcript on the real container, and — closing the caveat from the A/B comment above — natural termination on the serve path: with the model's own sampling (temperature 1.0, top_p 0.95 per generation_config.json), the gs64 container ends its turn on <|im_end|> after 214 tokens and the per-row container after 222, both limited=0. So the #455 non-termination symptom does not reproduce on Qwen on the EOS-aware path either. (Greedy decode in think mode does spiral — 2048 tokens of word-counting on the gs64 run — but that is the documented Qwen3 "do not use greedy" pathology, not a container property. Small gap worth noting: serve_sample implements temperature+top_p but not top_k, which upstream sets to 20.)

On your two questions: docs/qwen36.md is added in docs/inkling.md's structure — prebuilt containers as the easy path, converter as DIY, gs64 recommended per the A/B above with per-row still selectable. And the gs64 support from that A/B is now folded into this branch as promised (convert_qwen36.py --gs + grouped-scale expert GEMVs), so the engine can actually read the container the docs recommend.

Branch is 7 commits: engine, serve protocol (@minne100), gs64, docs, and the three fixes above.

[JustVugg]

Heads-up: #165 (DeepSeek V4 target-only CPU engine) has landed on dev, and this PR now conflicts. Same for #713. I resolved the conflicts #165 caused on several other PRs myself and pushed the merges to their branches, but I am deliberately not doing that here, and I want to be precise about why.

The conflict is additive in intent. Your render_chat_qwen and dev's new render_chat_v4 are two independent functions that happen to have been added at the same place in c/openai_server.py. Nothing is contested — both should exist.

But git's conflict boundaries interleave the two function bodies. The hunks cut across render_chat_qwen and render_chat_v4 mid-definition: one hunk ends at your parts = [] and dev's parts = [bos], the next mixes both message loops. Resolving it means reconstructing both functions by hand rather than picking sides, and a chat template that is subtly wrong does not fail loudly — it produces degraded output that looks like a model problem. That is not a call I should make inside someone else's template.

For you it is a couple of minutes: keep both functions whole, and make sure the dispatch covers qwen and deepseek_v4.

The rest is mechanical:

• c/coli — 4 hunks, the cmd_run / arch-dispatch region. dev added a deepseek_v4 branch beside yours.
• ARCH = "glm" — just the trailing comment listing the architectures; the union is glm | inkling | kimi | qwen | deepseek_v4.
• c/Makefile — resolves cleanly, no action.

One thing worth knowing since it cost three weeks here: your CI had also never run. The workflow sat in action_required, invisible from the PR page. I approved the blocked runs across the repo earlier today, so #712 and #713 now have real check results for the first time — worth a look before you rebase, in case they surface something.

[vladislavlad]

Hello!
What do you think about adding support for Qwen3.5-397B-A17B and Qwen3.5-122B-A10B?
What would I need to do to add support for these models?

[rofl0r]

What do you think about adding support for Qwen3.5-397B-A17B and Qwen3.5-122B-A10B?
What would I need to do to add support for these models?

afaik they (qwen 3.5 models) have the same architecture as qwen 3.6 , so they should mostly work as-is on this branch - they only need to be converted into a quantization format colibri understands. same holds for the qwen 3.8 series which will be released this week (crossing fingers they'll release also a 35B-A3B model... - the sweet spot for local inference)

[kreuzzelg]

B3 and the rebase are done, plus the rest of list C. Head is e755280 on current dev, 22/22 green, mergeable.

Including — and I did not expect this when I wrote that note — ARM (engines + NEON kernel exactness + tiny oracle), your #1083 job, which compiles qwen36.c on ARM for the first time and passes. The branch whose default we fixed is now built by the job that finding produced. Thank you for turning it into a class fix rather than a one-line patch on my file; the x = −128 contract you found on day one is the better half of that story.

Everything in the PR, against your review

	Item	Status	How
A1	is_attn sized from config.json, written from meta	✅	Reproduced your ASan signature first (heap-buffer-overflow, WRITE of size 1, load_meta:959), then refused: two files in one container disagreeing on the layer count is a broken container, not something to reallocate around
A2	7 unvalidated-config OOB sites	✅	One guard per dimension, kimi_k3.c shape. Each ceiling names the buffer that sets it. All seven verified to refuse: num_experts=2048, topk=512, dn_vdim=1024, dn_kheads=0, dn_convk=1, dn_conv_dim=999, n_layers=0
A3	dense tensor shapes unchecked	✅	Every dense tensor states the element count the forward pass indexes with, like f32_load
A4	token ids unchecked before the gather	✅	Bounds-checked; ids arrive from tokenizer, serve request and sampler — one of them remote
A5	expert_get fallback regression	✅	Wait-and-rescan verbatim from olmoe.c. Instrumented the old path first: at cap=2 it never fires, at cap=1 it does. Plus cap >= 1, or the new wait spins on the input that used to walk off slots[0]
B1	every chat request 400s	✅	FamilyLimits(8192, 262144, 1024, 8192, 1, 8, "Q36_MAXT") — the registry answered this while I was writing the question
B2	coli build KeyError	✅	build_target in the descriptor
B3	response_format desyncs the session	✅	capabilities.grammar_payload=False; the hardcoded tuple is gone from dev entirely
B4	no dispatch test coverage	✅	Tests iterate FAMILIES
B5	engine ships nowhere	✅	ENGINES, release build loop, dist/, SIBLINGS, make install — the registry coverage test enforces all five once the family is registered, and it is what caught the missing make install
B5b	the gate you asked for	✅	qwen36-tiny-check: token-exact at cache capacities 1/2/8, then ASan+UBSan with PILOT=1, plus a malformed-container step asserting the refusal and its reason
C	coli run → GLM template	✅	has_cli_adapter=False. cmd_run still dispatches per arch after that gate, so claiming an adapter we do not have would reinstate exactly the bug. The user gets "use coli chat or coli serve", which is true
C	cmd_stop, /v1/models, namespace over-claim	✅	process_names, default_model_id, exact model_types — qwen2/qwen3_moe/my_qwen_model no longer route here, asserted by the registry test
C	make install	✅	Added
C	make qwen36 CUDA=1 links -lcudart	✅	NOCUDA_CFLAGS, olmoe's shape; binary no longer links cudart in either mode
C	first_step/seen/momentum_logits	✅	Reset per request. They were per-process, so COLIBRI_RESIDENT=1 collected on request #1 and never again, and the router EMA carried one conversation into the next
C	--ram silently ignored	✅	Documented as unsupported: this engine needs full RAM residency, so there is no budget to size
C	g4/u4/d4 unread, +50% RSS	↗	Moved to #713 as you suggested — that is where they are read
C	tests/test_qwen36_ctx never runs	➖	Does run: TEST_RULES (c/Makefile:370) derives gates from build rules
D	resource_plan.py plans 0 bytes KV	✅	_qwen36_geometry — two terms, because only 10 of 40 layers hold a KV cache and the other 30 carry a fixed-size recurrent state. Your QWEN36_FIXTURE test specified it before I wrote it
D	no top_k in serve_sample	➖	Left in D on purpose: top_k is declared unsupported repo-wide (openai_server.py, Anthropic endpoint). Teaching it to qwen36 alone would make this engine inconsistent with every other and with the documented API
D	telemetry, KV prefix reuse, CANCEL, docs links	⬜	Follow-ups after merge

Not on your list, found on the way

Item	Why it matters
8192 stack overflow	float sc[8192] on the stack with Q36_MAXT as a second literal; raising the env moved the guard and not the buffer. Same class and fix as #110/#122
IDOT default-on	The one that became #1080/#1081/#1083
Oracle mode returned 0 on mismatch	The tiny gate could not have failed. Now match == n_new ? 0 : 1, as inkling.c does
make_qwen36_tiny.py emitted attention_only refs only	A gate built on that reference would never touch 30 of 40 layers. --ref-mode full added
The gate compared two different models	It went green then red on identical trees. The engine quantizes dense weights to int8 by default, the torch reference does not — 7 of 16 tokens, deterministic and identical at every cache capacity, which is also what ruled out an eviction bug. COLI_DENSE_I8=0 now, and the fixture is seeded: unseeded it passed three local draws and failed one in CI

Remaining from #1040: cuda_binary() checks GLM unconditionally, so --gpu auto fails for every sibling engine with CUDA linked. @sehHeiden found it; it is not qwen36-specific and I have not touched it here — say the word if you would rather have it in this PR than as its own.

[kreuzzelg]

Correction to my summary table: I marked g4/u4/d4 as "moved to #713" when they were still allocated here. That was a status I reported before doing the work, not a decision — my mistake, and thank you for a list precise enough that re-reading it caught me.

Done properly now in 67c7490: the packed int4 buffers are gone from this engine. It only ever read the unpacked int8 copy, so keeping them doubled expert-cache RSS on the very container the docs recommend. #713 restores them, which is where qt_note() and stage() actually read them — I verified that before removing rather than assuming it.

The gs64 container still loads and unpacks int4 to int8 for the CPU path; fixture still 16/16 at cap 1 and 8; suite green.

That closes list C apart from cuda_binary(), which is @sehHeiden's find and not qwen36-specific.

[kreuzzelg]

Where to get the model

Three pre-converted containers, all self-contained (weights + tokenizer.json + meta), Apache 2.0 like the upstream checkpoint:

Container	Format	Size	Use
Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64	int4, one scale per 64-weight group	~20 GB	recommended
Kreuzzelg/qwen36-35b-a3b-colibri-i4	int4, one scale per row	~20 GB	the A/B baseline
Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64	int4-gs64	22 GB	second model, same engine unchanged

hf download Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 --local-dir ~/Models/qwen36_i4_gs64
make -C c qwen36
COLI_MODEL=~/Models/qwen36_i4_gs64 ./c/coli chat

Or convert the bf16 checkpoint yourself (~70 GB download):

python3 c/tools/convert_qwen36.py --repo Qwen/Qwen3.6-35B-A3B --out ~/Models/qwen36_i4_gs64 --gs 64

Take gs64 unless you are reproducing an A/B. Controlled measurement on this model: cosine to the int8 anchor 0.98777 → 0.99313 and KL 0.109 → 0.080, i.e. ~44% less quantization error. The reason is moe_intermediate_size=512 — Qwen's expert rows are short, so per-row error concentrates the way it did on GLM in #455.

One caveat worth stating, because it cuts the other way on small cards: gs64 costs 1.69 MB/expert against per-row's 1.52, so the same VRAM budget holds ~10% fewer experts. On 8 GB that still came out ahead warm (11.55 vs 10.90 tok/s on a 3070) because accuracy outweighed the residency loss, but cold was ~8% worse. @sehHeiden measured the per-row container on a 6 GB 1660 Ti in #1040; whether the trade flips at that size is genuinely open, and the number to watch there is the [qtier] VRAM hit rate line rather than tok/s alone.

The KAT-Coder container is there because the engine takes any architecture-identical checkpoint unchanged — same config geometry, no code path of its own. It is a dev checkpoint, not a flagship; treat its outputs accordingly.

Also pushed just now (7e226fc), two stale references in docs/qwen36.md that this made me re-read:

• it linked docs/qwen36-cuda-tier.md, which is feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend #713's file and does not exist in this PR — that was the docs item in your list D;
• it still said the engine is picked because model_type contains qwen. Since fe3879c it is an exact match against the registry (qwen3_5_moe / qwen3_5_moe_text), which is precisely what stops this engine claiming qwen2_moe, qwen3_moe and every Qwen finetune — your list C item. The prose was describing the bug, not the fix.

[JustVugg]

Merged. Three weeks, the longest review in this repo's history, and it shows in what landed: an engine that refused to ship its own bugs — the A2 guards each verified to refuse, the oracle that could never fail fixed so it can, the gate that compared two different models caught and seeded, the IDOT default that became a repo-wide fix (#1080) and a new ARM CI job (#1083), and an RSS doubling caught by re-reading your own status table. colibrì's sixth engine, and the bar it was held to is now the bar.

Follow-ups, so nothing on the list gets lost:

• feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend #713 (CUDA VRAM tier) un-drafts on top of this — rebase when ready and it inherits front-of-queue
• cuda_binary() stays with @sehHeiden as its own PR, as agreed
• your D items (telemetry, KV prefix reuse, CANCEL, docs links) — file them as issues when you get to them and tag me
• the activation-hoist shape you asked to mirror: the x-side is perf: hoist expert activation quantization to layer level (bit-identical) #1071, the down-side half is perf(colibri): K3-lite — parallel silu + the down-side half of the #1071 hoist #1093 (in CI now) — both in colibri.c, both with the g_pq context pattern

The three HF containers in your comment are exactly what the release notes will point at. Thank you — this is what "model support" should mean.