[Feature]: Can you please consider adding qwen 3.8

[blenderman94]

Problem

Hello im really enjoying using kilibri so far and im wondered about if you maybe can add this model into the support list. https://huggingface.co/Qwen/Qwen3.8-27B i know we can run theese models with quantizations everywhere but it would be curious to run the full precision modell. thx in advance if you consider doing this. have a nice day and keep up the good work with kolibri.

Proposed solution

Alternatives considered

No response

Scope and compatibility

No response

[rofl0r]

qwen-3.8 27b is not a MoE model, so it's not really a target for this engine. however the 2.4T checkpoint is one, and should further MoE models from the family be released they should already be supported once #712 is merged.

[JustVugg]

Thanks for the kind words — and this is a genuinely interesting request, but the answer needs a distinction that decides everything.

I checked the model's config: Qwen3.8-27B is a dense model, not a Mixture-of-Experts one.

model_type:        qwen3_5
architectures:     Qwen3_5ForConditionalGeneration
num_hidden_layers: 64      hidden_size: 5120
MoE fields:        none (no num_experts / n_routed_experts / num_experts_per_tok)

That matters because colibrì's entire mechanism is streaming the experts a token actually routes to. In an MoE model ~5% of the weights are active per token, so we can keep the dense path resident and stream the rest on demand. In a dense model every weight is active on every token — there is nothing to stream selectively, and colibrì's core advantage evaporates. For dense models, llama.cpp and Ollama are already excellent and I'd genuinely point you there rather than pretend we'd be better.

But your actual goal — running it at full precision — is a real open thread here. Qwen3.8-27B in bf16 is roughly 54 GB, which is exactly the "RAM is the hard blocker" case. There is an open proposal for a streamable dense trunk (#826) precisely so colibrì can run where the dense part doesn't fit. If that lands, an unquantized 27B on a small machine becomes plausible. That's the issue to follow — I've linked this one there.

Two more things worth knowing:

• Its hybrid linear + full attention layer alternation is architecturally close to what the Qwen3.6 engine in review (feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712) already implements, so that machinery would partly transfer.
• If you want a Qwen MoE on colibrì today, feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712 (Qwen3.6-35B-A3B: 35B total / 3B active) is the one under active review.

Keeping this open as a tracked request, labelled model-support. I'm not going to promise a dense engine on a timeline — but I'd rather tell you exactly why it isn't a quick add than leave you waiting on something that was never going to happen.

[blenderman94]

thats a relief. im running theese things on a potato by today standards . (i even use the glm 5.2 too from hdd XD) fx 8350 , gtx 1080 32gb ddr3 . so we gladly welcome anyithing literaly . i almost thought i would have to make an engine on my own . but this is some nice news.

[JustVugg]

Update: #712 (qwen36 CPU engine) and #713 (CUDA VRAM tier) are both merged — the engine this thread was waiting on is on dev, GPU included.

Where that leaves this request:

• Qwen3.8-27B stays out of scope — it is dense (config above), and colibrì's whole mechanism is streaming the experts a token routes to; a dense model has no experts to stream. Quantized dense 27B is well served elsewhere.
• The Qwen3.8 2.4T MoE checkpoint is the interesting one. The qwen36 engine takes any architecture-identical checkpoint unchanged (it already runs KAT-Coder that way, zero code). Whether 2.4T qualifies is a config-geometry question.

@kreuzzelg — no obligation whatsoever, you have shipped an engine and a tier in one day: when you next have a quiet moment, would you check the 2.4T config against the registry entry (qwen3_5_moe exact-match, DeltaNet/gated-attention geometry, expert shape)? If it is architecture-identical, support is a registry line plus a container conversion — and at ~1.4 TB converted, the conversion itself is a community-scale effort we would coordinate rather than assume onto you. If the geometry differs, knowing where tells us the cost. @blenderman94 — a 2.4T MoE streamed from an HDD on an FX-8350 would be the most colibrì thing ever done; if the geometry checks out you will hear it here first.

[MrLafarge]

Thank you @JustVugg pushing this through! I would absolutely love to see Qwen 2.4T MoE supported down the line although I am not sure if its first entirely the philosophy of Colibri with its 95B active parameters (and its needs for certain hardware to do that). But I guess its for @kreuzzelg to decide.

[kreuzzelg]

@JustVugg @MrLafarge I'll first need to check whether my server is capable of running the Qwen 2.4T model with its 95B active experts. Honestly, that's the tricky part.

[kreuzzelg]

Did the geometry check. Short answer: the 2.4T config is architecture-identical — it passes every guard the engine has. But I have to be equally clear about the other half: my hardware cannot host it, by a factor of ~18, so I cannot be the one who validates a real run.

Everything below is my calculation, derived from the published config.json and measurements of my own 35B container. Please check it before anyone commits effort to this — I have not downloaded a single weight of the 2.4T checkpoint, so this is arithmetic, not observation.

Geometry: identical

Check	Registry / engine	Qwen3.8-2.4T-A95B
model_type	qwen3_5_moe, qwen3_5_moe_text	qwen3_5_moe_text	exact match
layer pattern	is_attn[i] = (i % 4 == 3)	full_attention_interval: 4, layer_types 92 entries → 23 full	identical
n_layers ≤ 512	guard	92	ok
n_experts ≤ 1024 (keep[])	guard	512	ok
topk ≤ 256 (idx[]/val[])	guard	10	ok
dn_vdim ≤ 512 (kvl[]/dl[])	guard	128	ok
dn_vheads % dn_kheads	guard	128 % 16	ok
q_heads % kv_heads (GQA)	guard	64 % 4	ok
dn_convk ≥ 2	guard	4	ok
max_context	262144	262144	exactly at the limit
rotary	35B: rotary_dim=64	partial_rotary_factor 0.25 × 256 = 64	identical
config_section="text_config"	falls back to root when absent	no text_config	handled

Parameter accounting from the config alone: routed experts 2370.8 B + shared 4.6 B + attention 9.6 B + linear-attn 20.8 B + embed/lm_head 4.1 B = 2.410 T against an announced 2.4 T. That the two agree is my main reason for trusting the rest of the arithmetic.

Size — your ~1.4 TB estimate holds

At my converter's measured cost (int4 experts with F32 group scales = 4.56 bits/weight; dense, attention and embeddings stay F16 at 2 B):

container ≈ 1.42 TB = 1.29 TiB

Cross-check: Qwen's own NVFP4 release is 1.32 TiB and MXFP4 1.45 TiB. Landing between them is what I would expect, and it says the format is not the problem.

KV, via the registry's own _qwen36_geometry on this config: 1.44 GiB at 8k, 11.5 GiB at 64k, 46.0 GiB at 256k (23 of 92 layers only). The DeltaNet recurrent state is 0.55 GiB flat, at any context — the hybrid earning its keep.

Why it is not me who runs it

Container	1.42 TB
Peak RSS (measured ratio 1.150 on my 35B run: 21.45 GiB container → 24.66 GiB RSS)	1.64 TB
+ KV at 256k	49 GB
Total	~1.69 TB
My machine	94 GB RAM, 2.6 TB free disk, 2× 8 GB VRAM

Factor ~18 on RAM. And it is structural, not a tuning matter: the qwen36 warmstart deliberately loads all experts into RAM, because the alternative was a ~12 ms container read landing mid-decode (that change took a single-GPU run from 139 ms/token to usable). A 2.4T checkpoint needs the opposite design — GLM's disk tier, streaming per routed expert.

Even granting that tier, an upper bound from my disk: 95 B active weights ≈ 26–50 GB read per token; measured 3.30 GB/s (16 parallel streams, iflag=direct, cache bypassed) → 8–15 s/token. Which, to be fair to @blenderman94's spirit, is exactly the "2.4T MoE streamed from an HDD" scenario — just not something I can present as a benchmark.

Disk is the one resource that is not the blocker: 1.42 TB fits in 2.6 TB. But BF16 at 4.45 TiB does not, so even converting here would mean streaming from a 4-bit source, i.e. requantizing already-quantized weights, and I have no measurement of what that costs.

Two things worth deciding regardless of the 2.4T question

1. The registry already claims it. qwen3_5_moe_text is in my descriptor's model_types, so a 2.4T checkpoint dropped in today resolves to the qwen36 engine and displays as "Qwen3.6-35B-A3B / 35B". Whatever is decided about support, that string is wrong for this checkpoint. A second descriptor, or a size-aware display, either works.
2. mtp_num_hidden_layers: 1. The checkpoint carries one Multi-Token-Prediction layer the engine knows nothing about. Probably just tensors to skip, but the converter's inventory would have to skip them deliberately rather than by accident.

What I have not checked

Tensor naming. expert_inventory matches model.layers.<i>.mlp.experts.<j>., and I have only the config, not the weight index. If the 2.4T checkpoint names its experts differently, that regex is the first thing to fail — and it is a five-minute check for anyone who has the index file.

Happy to write the descriptor and validate it against a synthetic tiny model with this exact geometry (92 layers, interval 4, 512 experts, topk 10) — that path needs no weights at all and would catch a config-shape error before anyone spends a terabyte of bandwidth. What I cannot do is the real conversion or the real run.

[blenderman94]

curious lookin. i have a 2tb hdd and time. i got used to waiting for glm this cant be much worse either. XD how do i can try this? just to clear how big a block will be i have to hold to perform an imediate result? betwen layers or ops? maybe i write an engine for if i get realy bored (what im mostly doing :) ) but one thing i have to know is the big model is realy so much better then the 27b variant to waitbpossibly a week for a sentence to get ?

[kreuzzelg]

@JustVugg — geometry check done, and the answer to your question is the good one: architecture-identical. Every guard the engine has passes (full table in my comment above): qwen3_5_moe_text model_type, is_attn[i] = (i % 4 == 3) matches the 92-layer / interval-4 pattern (23 full-attention layers), 512 experts / topk 10 / dn_vdim 128 / GQA 64:4 all inside limits, rotary_dim resolves to 64 exactly like the 35B. Parameter accounting from the config alone lands at 2.410 T against the announced 2.4 T, which is my main reason to trust the rest of the arithmetic. So by your own formula: support is a registry line plus a container conversion — except the registry line already exists, which is its own small problem:

1. The registry claims this checkpoint today. qwen3_5_moe_text is in the qwen36 descriptor's model_types, so a 2.4T container dropped in right now resolves to the qwen36 engine — and displays as "Qwen3.6-35B-A3B / 35B". Whatever we decide about official support, that string is wrong for this checkpoint. A second descriptor or a size-aware display, either works; happy to write it.

2. One correction to my own numbers above — the RAM conclusion, not the arithmetic. My "factor ~18" describes the current qwen36 warmstart (all experts resident), not the model. I said a 2.4T checkpoint needs the disk-tier design instead, but failed to compute what that design actually needs resident: attention + DeltaNet + embed/lm_head (34.5 B, F16) ≈ 69 GB, shared experts int4 ≈ 2.6 GB, staging for 10 routed experts/layer ≈ 1 GB, KV@8k + DeltaNet state ≈ 2 GB, overhead ≈ 5 GB → ~80 GB floor, ~45 GB with COLI_DENSE_I8. So with a GLM-style tier, RAM is not the blocker — any 64–96 GB box qualifies, mine included. The honest blocker is disk bandwidth. (Caveat on my KV numbers: they assume F32 cache entries; if the engine stores KV in F16 they halve — worth one look at the alloc path before anyone quotes the 46 GiB @ 256k.)

3. What tok/s to expect — a range, not a number. My 8–15 s/token was a cold-cache upper bound (26–50 GB read per token at my measured 3.3 GB/s, single NVMe, no RAID). It assumes zero expert reuse between tokens; on the 35B the cache hit rate is 95%. If the 2.4T routes with similar locality, effective reads drop by an order of magnitude and NVMe could plausibly land around ~0.5–2 s/token. Nobody has that measurement yet — it's the first interesting number a real run would produce. @blenderman94, in the spirit of full honesty about the HDD dream: at ~150 MB/s even a 95% hit rate leaves you at minutes per token. NVMe makes this plausible; spinning rust makes it performance art.

4. Two five-minute checks that need no weights and no download, if anyone gets there before me:

• model.safetensors.index.json from the HF repo against the converter's expert_inventory regex (model.layers.<i>.mlp.experts.<j>.). If the 2.4T names its experts differently, that regex is the first thing to fail.
• The checkpoint carries mtp_num_hidden_layers: 1 — one Multi-Token-Prediction layer the engine knows nothing about. Probably just tensors to skip, but the converter's inventory must skip them deliberately, not by accident.

5. Conversion logistics. BF16 source is 4.45 TiB — that needs someone with ~5 TB free, which rules my 2.6 TB out. The alternative is converting from Qwen's own NVFP4/MXFP4 release (1.32/1.45 TiB, fits), but that means requantizing already-quantized weights, and the quality cost of that should be measured on the tiny fixture before anyone spends the bandwidth.

Timeline, so nothing silently stalls: I'm mostly offline for the next few weeks. The synthetic tiny-model validation (92 layers / interval 4 / 512 experts / topk 10 — needs zero weights, catches any config-shape error before anyone spends a terabyte) plus the descriptor/display fix and the MTP skip are exactly the scope I'd pick up when I'm back, if nobody has claimed them by then. If someone wants them sooner, go ahead — the geometry table above plus the two checks in point 4 are everything needed to start, and I'll gladly review.

[JustVugg]

Confirmed point 1 against the code, and you are right.

qwen36 declares model_types=("qwen3_5_moe", "qwen3_5_moe_text") with a hardcoded display_name of "Qwen3.6-35B-A3B". So the registry does claim any qwen3_5_moe_text checkpoint today, and would announce a 2.4T model as a 35B one. That is worth fixing on its own merits, independently of whether Qwen3.8 is ever supported: a loader that misreports which model it loaded is the kind of thing that wastes someone a day.

The rest of your analysis is the most useful model request this tracker has had, and the reason is point 2. You recomputed your own conclusion and reversed it. The corrected number is also the one that changes the answer: ~80 GB resident with a GLM-style disk tier, ~45 GB with COLI_DENSE_I8, which moves this from impossible to merely expensive.

On your KV caveat, worth checking the alloc path as you say. It is also the exact ground GLM-5.3 just covered: absorbing kv_b_proj into the two ends leaves the cache holding the 512-wide latent instead of expanded keys, 1.39 MB per token down to 33 KB. If Qwen3.8's attention has the same shape, that lever exists rather than needing inventing.

Your scope is the right scope and it is yours if you want it when you are back. The synthetic tiny fixture first is exactly right: 92 layers, interval 4, 512 experts, topk 10 catches every config-shape error for the cost of no weights at all, and it is how GLM-5.3 was built. The MTP skip should be deliberate too, and there is precedent to copy: the GLM-5.3 converter classifies all 92 tensor kinds explicitly and stops on an unrecognised name, because a converter that silently skips what it does not know produces a checkpoint that loads and is quietly missing a tensor.

Nobody has claimed it. Thanks for the honesty about the timeline, and for telling @blenderman94 the truth about spinning disks rather than the encouraging version.

[blenderman94]

well nice job guys you semingly care about what people say what is nice. Because you did like that i tell you that im working on a solution on my own as well with my methods (just straight out of curiosity and aging hardware) but i genuinely belive we not run llms today optimaly . but i have to see or fail that myself by making someting like lama cpp by hand out of noting.

[gitmeatarru]

Thanks for the kind words — and this is a genuinely interesting request, but the answer needs a distinction that decides everything.

I checked the model's config: Qwen3.8-27B is a dense model, not a Mixture-of-Experts one.

model_type:        qwen3_5
architectures:     Qwen3_5ForConditionalGeneration
num_hidden_layers: 64      hidden_size: 5120
MoE fields:        none (no num_experts / n_routed_experts / num_experts_per_tok)

That matters because colibrì's entire mechanism is streaming the experts a token actually routes to. In an MoE model ~5% of the weights are active per token, so we can keep the dense path resident and stream the rest on demand. In a dense model every weight is active on every token — there is nothing to stream selectively, and colibrì's core advantage evaporates. For dense models, llama.cpp and Ollama are already excellent and I'd genuinely point you there rather than pretend we'd be better.

But your actual goal — running it at full precision — is a real open thread here. Qwen3.8-27B in bf16 is roughly 54 GB, which is exactly the "RAM is the hard blocker" case. There is an open proposal for a streamable dense trunk (#826) precisely so colibrì can run where the dense part doesn't fit. If that lands, an unquantized 27B on a small machine becomes plausible. That's the issue to follow — I've linked this one there.

Two more things worth knowing:

• Its hybrid linear + full attention layer alternation is architecturally close to what the Qwen3.6 engine in review (feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712) already implements, so that machinery would partly transfer.
• If you want a Qwen MoE on colibrì today, feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712 (Qwen3.6-35B-A3B: 35B total / 3B active) is the one under active review.

Keeping this open as a tracked request, labelled model-support. I'm not going to promise a dense engine on a timeline — but I'd rather tell you exactly why it isn't a quick add than leave you waiting on something that was never going to happen.

Thank you for the explanations.

I wish you success on this new path that no one has ever taken before!

We’ll be looking forward to new releases!