[Feature]: Kimi K3 Original Weights loading

[MichaelFomenko]

Problem

Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility.

Proposed solution

We need the loading of the Original Kimi K3 Weights, without any changes. The Engine should work out of the box with the Original Quantization. Because any Changes to the Original Weights will degrade the Quality. Its not the Situation like in GLM 5.2 where Quantization makes sense, but not in Kimi K3.

Alternatives considered

No response

Scope and compatibility

No response

[JustVugg]

You're right on the core point, and it's an important distinction: K3's routed experts are QAT-trained at MXFP4, so re-quantizing them (int4 per-row, or even our gs64) would fight the training and lose quality. For the experts, the correct thing is exactly what you ask — load the native MXFP4 blocks and stream/decode them as-is, no re-quant. That's the plan.

One honest nuance from K3's own config, so we set expectations correctly. K3 is a mixed checkpoint, not uniformly MXFP4. Its quantization_config ignore-list keeps these in bf16:

• self_attn (MLA: q_lora 1536 / kv_lora 512)
• shared_experts + first-k-dense mlp
• lm_head, vision tower

Only the routed experts (896/layer, 16 active) are MXFP4. So the split for colibrì is:

• Experts → keep native MXFP4, streamed from disk, decoded on the fly (this is exactly the path we already stream int4 experts through; it needs one new decoder: MXFP4 = e2m1 elements + shared e8m0 scale per group of 32).
• Dense (bf16) → this is the catch. The dense weights are ~56 GB in bf16. colibrì's whole reason to exist is small-RAM boxes (the dense tensors are the part that stays resident), and 56 GB won't fit there. The dense is not QAT'd, so quantizing it to int4 is the same safe trade-off as GLM-5.2 — no QAT to fight. On a large-RAM machine you could keep dense bf16 untouched; on a 24–32 GB box the dense would need int4 while the experts stay native MXFP4.

So the resolution isn't "load everything untouched or nothing" — it's native MXFP4 for the experts (no changes, as you want), dense left bf16 where RAM allows and int4 only where it must fit. That preserves the QAT quality exactly where it matters.

Status: this is on the roadmap and next in line. We're finishing an Inkling conversion first, then K3 is the next model through the pipeline (convert → native MXFP4 experts + dense tiering → quality test). Labeled enhancement / model-support. I'll link the MXFP4 decoder work back here when it lands.

[MichaelFomenko]

Maybe for people with low ram, they can convert the dense part of the Model to Int4 or int8 and load only this extra Safetensor file to the Original Weights Folder when low ram usage is needed. Or the extra Safetensor int4 or int8 can be set by extra dense_directory. Or just convert the original weights on the fly by loading from the original weights to ram, because the dense part will stay in ram and not streamed from the disk.

[Sarajevo67]

It would be great, if Colibri dev team use original K3 safetensors files, 96 of them (as in the july 27th release), including dense part, as it is.
That would attract much larger community, including those who downloaded weights "just in case..." and would like to see, what that beast, residing on the SSD - can do?

From my GLM 5.2 Colibri experience, it is painful to realize that 384 GB of weights, you downloaded yesterday, suffer from quantization errors, that obliterate quality benchmarks and you need another g64 429 GB download. When I think ,that it could happened with 1.56 TB DL, I don't feel enthusiastic about it.

Related to the shared (dense) experts. Keep it on the disk, as they are and, in the subdirectory, forge optional quantizations.

Please, keep in mind, the real reason, why people use Colibri?
It's not the speed.
It's not ability to claim: look I'm using GLM 5.2 (or Kimi K3).
It's ability to get BEST possible answer, from your local computer. In the hour, day, week... Doesn't matter.

[JustVugg]

Delivered in v1.3.0. #676 (@steve-m, with the chat/API/Web integration by @ZacharyZcR) adds c/kimi_k3.c: the QAT-trained MXFP4 routed experts are streamed straight from the original Hugging Face shards, never re-encoded — exactly what you argued for here. The bf16 dense set is quantized at load time (K3_BITS, default int4-g64), so the parts that are not QAT'd are the only ones touched.

Tokenizer validated at 77/77 token ids against Moonshot's official encoding_k3.py. Full-model generation still needs a host with ~1.6 TB for the snapshot — nobody has run it end to end yet, and the release notes say so. If you have that hardware, we'd very much like your report.

Closing as implemented; reopen if the loading path doesn't match what you expected.