[Feature]: Support dual-quantization (ultra-low bit experts + high-bit attention)

[RaphyTwin]

Problem

Loading 4-bit experts from disk still transfer too many gigabytes per token.

Proposed solution

Implement a mixed-precision quantization scheme:
Keep crucial shared layers (attention, input embeddings) at 4-bit or 8-bit permanently in VRAM/RAM.
Quantize the dynamically loaded MoE experts down to ultra-compressed 1.5-bit or 2-bit weights (using HQQ or QuIP) to drastically minimize disk-transfer size.

Alternatives considered

No response

Scope and compatibility

(Technical requirements:)
Dequantization must happen on-the-fly in CPU registers/L3 cache during forward pass to avoid memory bandwidth bottlenecks.

[JustVugg]

Solid instinct, and it aligns with two things already moving:

• Converter: per-tensor-type mixed precision (int8 for shared/o_proj/kv_b_proj, int4 rest) #243 (open, in review) adds per-tensor-type mixed precision: int8 for shared_experts/o_proj/kv_b_proj, int4 for the rest — i.e. keep the attention/shared path higher-bit, quantize the routed experts harder. That's the 'high-bit attention, low-bit experts' half of this.
• Group-scaled int4 (fmt=4): one scale per 128 elements — fixes incoherent output #242 (merged) + int3-g64 (fmt=5): per-group-scale 3-bit weights, the size/quality point your own ablation (#132) says the next container should sit at #168 add group-scaled int4/int3 (fmt=4) which lets the routed experts go lower-bit without the coherence collapse that per-row int4 causes (see issue_grouped_quant.md).

The combination you're describing — ultra-low-bit (int2/int3-g64) routed experts + int8 attention/shared — is a real container design point. Keeping open as the umbrella; #243 is the first concrete step. If the grouped-quant A/B (pending the FP8 download) shows int3-g64 experts hold quality, this becomes very attractive on low-RAM hosts.

[JustVugg]

This is exactly the lever #153 points at from the other side: their data shows int4 IDOT costs 9-18% perplexity on the attention projections while the experts tolerate int4 fine. So dual-quantization — ultra-low-bit experts (int4/int3-g64) + higher-bit attention (int8/bf16 dense, which is already resident and cheap in bytes since it's only ~10 GB) — is the principled fit. The converter already supports per-projection precision (--dbits for dense, expert bits separate), and gs64/fmt=5/fmt=6 give the low-bit expert side. What's missing is the default policy that keeps attention at higher precision automatically. Worth wiring: it's mostly a converter default + a doc, and it directly recovers the #153 perplexity. Linking #153/#452 here.

[JustVugg]

This shipped — through the converter rather than as a runtime knob, which turned out to be the right layer for it.

c/tools/convert_fp8_to_int4.py:204-217 classifies shared / o_proj / kv_b_proj / attention / dense-MLP / routed tensors separately, and :437-467 exposes --ebits --xbits --attn-bits --o-bits --kvb-bits --shared-bits --dmlp-bits. The engine resolves format and group size per tensor (c/colibri.c:4347) and budgets mixed widths (:8182, covered by c/tests/test_cap_mixed_width.c). :532 prints the resulting precision map.

So "ultra-low-bit experts + high-bit attention" is exactly what you can build today. One honest caveat: only --xbits is documented in docs/SETTINGS.md:74 — the per-matrix flags aren't in the docs anywhere yet, which is a real gap I'm tracking separately. Closing as delivered.