fmt=5: a deployable E8-lattice grouped container — int3 that beats shipped int4 quality at 25% fewer bytes (design + plan)

[ZacharyZcR]

Why now

Decode on every measured host is a bandwidth wall: expert bytes ÷ effective GB/s is the largest bucket on the critical path (6×5090 forensics in #431: ~47 ms/token of CPU-tier expert sweep at hit 100%). Bytes are speed. The quantization ablations (#81, #255, #347) found the lever:

container	quality (MMLU-class Δ)	bytes vs int4
shipped per-row int4	−9.3pp	1.00×
uniform int3-g64	−7.5pp	0.75×
int3-g64-e8-rot	−5.9pp	0.75×

A 3-bit E8-lattice container measures better than the shipped 4-bit while carrying 25% fewer bytes — that is simultaneously: −25% on the CPU-tier wall, +33% experts per GB of VRAM (the whole routed set of GLM-5.2 drops to ~280 GB), and a quality gain over what users run today.

But those numbers are simulated: tools/quant_ablation.py quantizes to the nearest E8 point and stores floats. There is no deployable index encoding, no container format, no decode kernel. This issue is the plan to build them.

The pedigree (so nobody thinks this is exotic)

E8 codebooks are QuIP#'s core trick (ICML'24) and are mass-deployed in llama.cpp's IQ2/IQ3 quant family (E8-derived grids + sign-bit factoring, MIT). Our additions are the rate-scaled ball (#347 — the radius must grow with the bit budget), the g64 outer scale, and the target: a tiered MoE expert stream where bytes convert to tok/s at a measured rate.

Design: fmt=5 (E8 grouped container)

1. Block encoding — decision point, two candidates:
   • (a) IQ3-style subset codebook (adopt llama.cpp's proven construction, MIT): small LUT (256–512 entries) that fits in SIMD registers / GPU shared memory, signs factored out by parity. Deployable today, kernels are a known quantity.
   • (b) our rate-scaled ball with structured coordinate coding: matches the ablation exactly but needs entropy-shaped index construction (a 3-bpw E8 ball has ~10⁷ points — naive indexing doesn't close). Research-grade.
   • Plan: (a) first, re-validated through the ablation harness (swap _quant_e8's ball for the candidate codebook, OLMoE A/B in hours, then GLM n=200). (b) stays a research fork if (a) leaves quality on the table.
2. Container layout: per tensor — packed block indices + sign words, [O, ng] fp32 outer scales (g64, same shape discipline as fmt=4), rotation seeds (rotations regenerate deterministically from seed — nothing stored beyond a per-tensor seed). Rider tensors (.qs shape) keep the existing shard/index machinery.
3. Engine dispatch: QT gains fmt=5; matmul_qt routes to new CPU kernels (AVX2/AVX-512: LUT gather + sign-apply + FMA, group scale exactly as matmul_i4_grouped); CUDA rides the cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334) #451 GroupDesc path — the per-group scale plumbing is already merged-ready, the g4 kernels get e8 siblings with the LUT in shared memory.
4. Converter: port _e8_nearest/rotation from the ablation into convert_fp8_to_int4.py with index emission; --ebits 3 --e8 on the existing mixed-precision flag surface (MTP head stays int8, io/shared tensors keep their existing overrides).
5. Placement arithmetic (the payoff): ~280 GB total → on a 6×5090/251 GB host, VRAM 176 GB holds ~12.4k experts (+34%) and RAM covers the rest twice over; on 128 GB hosts the pinned share rises ~33%.

Validation ladder (each step gates the next)

1. Codebook A/B in the ablation harness (OLMoE, torch, hours) — candidate (a) vs the published ball numbers.
2. Round-trip exactness oracle: encode→decode == the torch dequant, bit-for-bit (the fmt=4 oracle pattern, cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298).
3. CPU kernel oracle vs reference; CUDA oracle via the cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334) #451 test pattern.
4. GLM full convert + n=200 quality three-suite vs the int4 baseline (baseline run in flight).
5. Speed A/B on the 6×5090 host: same protocol as this week's forensics — P0-EXEC GB/s fingerprint, usage-snapshot control, warm-segment rates.

I'll take the workstream; kernels and converter in increments, same discipline as the #431 series. Input welcome on the codebook decision (1) — especially from anyone who has run the llama.cpp IQ3 grids up close.

[ZacharyZcR]

Validation ladder step 1 complete — codebook decision: candidate (a), rotation mandatory

Ran the six-scheme A/B on OLMoE-1B-7B (n=200 × hellaswag/arc_challenge/mmlu, same harness as the published ablations — the first four rows reproduce them exactly, which is the harness-integrity check):

scheme	MEAN acc_norm	Δ vs fp16
fp16 (ceiling)	58.0%	—
int4 per-row (shipped container's scheme)	48.7%	−9.3pp
int3-g64 uniform	50.5%	−7.5pp
int3-g64-e8-rot (our rate-scaled ball, simulated)	51.5%	−6.5pp
int3-iq3 (IQ3_XXS codebook, faithful: parity-constrained signs, fp16 super + 4-bit sub scales)	49.3%	−8.7pp
int3-iq3-rot	51.5%	−6.5pp

Two findings:

1. The deployable IQ3_XXS codebook + rotation exactly ties our simulated ball — 51.5% vs 51.5%. Candidate (a) leaves no quality on the table, and it arrives with a mass-deployed encoding: 256-entry 4-dim magnitude grid + parity-factored signs + sub-scales, LUT small enough for SIMD registers / GPU shared memory. fmt=5 adopts the IQ3-style block structure. Candidate (b) (structured coding of the rate-scaled ball) is closed as unnecessary.
2. Rotation is where the lattice earns its keep: the same codebook without rotation loses 2.2pp (49.3 vs 51.5) and barely beats shipped int4. The rotation seeds are part of the format, non-optional.

The -iq3 scheme (grid table extracted from ggml-common.h, MIT) is in tools/quant_ablation.py for reproduction. Next per the ladder: the round-trip index encoder (step 2) and the fmt=5 container spec, then CPU/CUDA decode kernels on the #451 plumbing.

[JustVugg]

Tooling is moving: #453 (IQ3_XXS-codebook scheme in the quant ablation, settles the codebook decision) and #458 (fmt=5 index codec — the deployable bytes) are the two steps toward this. Once the ablation picks the codebook and the index codec is verified, the fmt=5 E8-lattice container becomes packable. Keeping this open as the design/plan umbrella. Strong direction — int3 that beats shipped int4 at −25% bytes is exactly the disk-streaming lever that matters here.

[ZacharyZcR]

Design constraint found while starting the converter: where the rotation lives

The ablation settled that rotation is mandatory for this codebook (2.2pp — #453). Rotation means the container stores W_rot = W @ Q, and inference must compute

y = W @ x = W_rot @ (Qᵀ x)

so every matmul needs its activations transformed first. Q = D·H/√n (random signs + Hadamard), so the transform is a fast Walsh–Hadamard transform — no matrix is stored, and the seed regenerates D deterministically.

Measured the cost directly (single core, -O3 -march=native, GLM dims padded to powers of two: 6144→8192, 2048 exact):

	per call
FWHT on the hidden dim (8192)	8.1 µs
FWHT on the expert-inter dim (2048)	2.1 µs

The placement matters more than the kernel does:

• naive (inside the expert matmul, i.e. per expert per projection): 75 layers × 8 experts × 3 projections → 11.0 ms/token
• correct (once per layer, since all 8 routed experts of a layer consume the same input row): 1.4 ms/token

Against the ~47 ms/token CPU-tier expert sweep that #431's forensics identified as the wall, the amortized figure is ~3% — acceptable for a container that removes 25% of the bytes. The naive placement would eat a quarter of the win.

So the format's contract is: the engine rotates activations once per (layer, projection group), and fmt=6 kernels consume already-rotated activations. The gate/up projections share one transform of the layer input; down needs its own on the intermediate. Writing it down here because it is invisible from the kernel signature — an implementer who puts the FWHT inside matmul_e8 gets a correct but 8× more expensive engine.

Follow-ups this implies, before the converter can emit fmt=6 weights:

1. fwht_rows() in the engine (scalar first, then SIMD — it is a memory-light log-n pass)
2. rotation seed in the container metadata (per tensor; regenerated, never stored)
3. dimension padding rule for non-power-of-two dims (6144 → 8192) decided and tested — this is the piece I want to get right before generating a 280 GB checkpoint

[ZacharyZcR]

Padding decision for non-power-of-two dims — and a caveat it exposes for GLM

The FWHT rotation needs a power-of-two dimension. GLM's hidden is 6144 (down_proj input is 2048, which is fine). Tested five strategies on a realistic heavy-tailed 6144-wide tensor (3% outliers), measuring the fmt=6 encode/decode rel-RMSE — what the container actually stores:

strategy	rel-RMSE	byte cost
no rotation	0.2092	1.00×
A: pad 6144→8192, rotate globally	0.1686	1.33× (33% waste)
B: block-diag 4096+2048	0.1928	1.00×
D: block-diag 2048×3	0.1929	1.00×
E: block-diag 1024×6	0.1932	1.00×
F: block-diag 512×12	0.1935	1.00×
G: pad-rotate, truncate storage to 6144	0.4614	1.00×

Three findings:

1. Block size barely matters (0.1929–0.1935 across 2048/1024/512) — all the quality of block-diagonal rotation is there at the coarsest split, so the cheapest FWHT (largest blocks) is free to use.
2. Global rotation's quality edge is inseparable from its byte cost. A stores 8192 indices for 6144 weights: 3.0625 bpw × 1.33 = 4.07 bpw — identical to the shipped per-row int4 it was meant to beat. The whole point of fmt=6 is −25% bytes; padding gives that back. Truncating the padded rotation to recover the bytes (G) destroys orthogonality and wrecks quality (0.46). So on 6144-wide tensors, block-diagonal is the only rational choice.
3. This means GLM will not reach the OLMoE ablation number. tools: IQ3_XXS-codebook scheme in the quant ablation — settles #452's codebook decision #453's 51.5% (−6.5pp, ties the E8 ball) was measured on OLMoE, whose dims are 2048 — a power of two, so it got global rotation. GLM's 6144 tensors are stuck with block-diagonal, which recovers roughly half the rotation benefit (0.193 vs 0.169 vs 0.209 no-rot). The down_proj (2048) still gets global rotation; only the gate/up (6144-input) are limited.

Decision: fmt=6 uses per-tensor block-diagonal rotation with the largest power-of-two blocks that tile the input dim (6144 → 2048×3, 2048 → one 2048 block). Blocks and their seeds are metadata; the engine's FWHT runs per block.

Honest consequence to measure before committing to a 280 GB convert: the GLM quality delta needs its own n=200 run — the OLMoE −6.5pp is an upper bound, not a promise, for the 6144-wide projections. That measurement is the real gate on whether fmt=6 ships for GLM or stays an OLMoE-class win. Better to find the number now than after the conversion.

[ZacharyZcR]

Attempted the GLM quality gate on NVFP4 weights — source contamination, result void (methodology note)

Tried to get GLM's fmt=6 number without the raw fp16 checkpoint (not available), using the on-hand GLM-5.2-NVFP4 as a proxy source: dequantize real expert tensors from it and measure fmt=6 vs shipped-int4 reconstruction rel-RMSE. Result, on 24 real GLM gate/down tensors:

scheme	rel-RMSE
fmt=6 block-diag rotation	0.1935 ± 0.0000
shipped int4 per-row	0.1554 ± 0.0053

Taken at face value fmt=6 loses by 24.5% — but the result is invalid, and the ± 0.0000 says why. Twenty-four different tensors cannot give identical fmt=6 error unless the input distribution is itself discrete. It is: NVFP4 already quantized these weights to FP4, so a dequantized tensor carries only 16 distinct magnitudes (the FP4 codebook × scale), not a continuous weight distribution.

Comparing 3-bit (fmt=6) against 4-bit (int4) on already-near-4-bit values measures "which scheme better re-encodes FP4", not "which preserves the original weights" — and 4-bit trivially wins by being closer to the source's own precision. The measurement is contaminated by the source, not informative about the containers.

So:

• The OLMoE −6.5pp (51.5% = the E8 ball, tools: IQ3_XXS-codebook scheme in the quant ablation — settles #452's codebook decision #453) stands — that used a genuine fp16 model.
• The GLM fmt=6 quality gate remains open. NVFP4 is confirmed not a usable proxy source; the block-diagonal-rotation caveat from the padding probe (fmt=5: a deployable E8-lattice grouped container — int3 that beats shipped int4 quality at 25% fewer bytes (design + plan) #452 above) still means GLM should be expected to land somewhere between no-rotation and the OLMoE global-rotation number, but the exact figure needs the raw GLM-5.2 fp16/bf16 weights — which aren't on the bench (official HF 403'd earlier).

Marking this path closed so nobody re-runs it. The engine-side work (fmt=6 kernel #465, codec #458, FWHT with the placement rule) is independent of this gate and stands on its own; the gate decides shipping a GLM checkpoint, not whether the format is correct.

[JustVugg]

Step 3 landed: #465 (fmt=6 E8/IQ3 lattice container — CPU decode kernel + dispatch) is merged into dev, on top of #453 (codebook ablation) and #458 (index codec). So the container is now decodable on the CPU path. Nice arc — int3 that beats shipped int4 at −25% bytes is exactly the disk-streaming lever that matters here. What's the next step you'd like to see (packer wiring, quality sweep on a real container)?

[ZacharyZcR]

Packer wiring first — it's the dependency for everything else. The sweep needs a real container to sweep, and the CPU kernel (#465) is currently only exercised by fixtures. Concretely, in order:

1. Converter → fmt=6 container (convert.py wiring on top of tools: fmt=5 index codec — the deployable bytes for the E8/IQ3 container (#452 step 2) #458's codec, block-diagonal FWHT per the placement rule in quant.h). One flag: --expert-bits e8.
2. Quality sweep on OLMoE, not GLM, and that's deliberate: OLMoE has true fp16 weights, so the ablation numbers (iq3-rot −6.5pp vs int4's −9.3pp at −25% bytes) are cleanly attributable. That validates the pipeline end-to-end: converter → container → engine: fmt=6 E8/IQ3 lattice container — CPU decode kernel and dispatch (#452 step 3) #465 decode → eval.
3. GLM container is gated on sourcing real fp16 — the NVFP4 repack is already-quantized (16 discrete magnitudes per tensor; measured std ±0.0000 on the dequant residual), so any quality number derived from it measures source pollution, not the format. If anyone has a line on the original bf16/fp16 checkpoint, that unblocks the headline artifact.
4. After that, the CUDA sibling kernel rides cuda: grouped-int4 (fmt=4) support in the expert-group kernels — opens the GPU tier to g64 and E8 containers (#334) #451's GroupDesc plumbing — the dispatch seam is already there.

One caveat worth stating up front: GLM's hidden dim is 6144, not a power of two, so the rotation is block-diagonal (3×2048) rather than global — on OLMoE the global-rotation variant was worth the full 2.2pp, and the block-diagonal ablation came in slightly behind (rel-RMSE 0.193 vs 0.169). The GLM number will land somewhere between int4 and the OLMoE headline; the sweep in step 2 is what tells us where.

I'll open the converter PR next.