qwen36: vectorise the int4 expert unpack (2.10x CPU decode, bit-exact)

[lornecodes]

Every expert cache miss in qwen36.c unpacks a packed-int4 expert to int8 in the slot, and the loop doing it was indexed by element:

for (i = 0; i < want_w; i++) {
    uint8_t byte = raw[i >> 1];
    int8_t v = (i & 1) ? ((byte >> 4) & 0xF) : (byte & 0xF);
    if (v & 8) v -= 16;
    s->g[i] = v;
}

want_w is 3 * inter * hidden = 6,291,456 for Qwen3.6-35B-A3B — 6.29M iterations per miss, each reloading raw[i>>1], doing an i&1 select, and taking a branch.

Two changes, in order. Walking bytes and sign-extending by shifting removes the branch. Vectorising then needs an interleaving store, because the two nibble streams land consecutively in the output — which is why no compiler reaches it from the scalar form. Checking the disassembly after the branchless rewrite confirmed zero vector registers in the loop. Written explicitly it emits vst2q on NEON and vpunpcklbw/vpunpckhbw on AVX2, with the scalar form kept as the tail and the portable fallback.

Bit-exactness

Integer throughout, so unlike the float reductions in #442 there is no reassociation question — this is exact by construction rather than by tolerance. In vectors the signed value of a nibble n is (n ^ 8) - 8; the scalar tail gets the same result by casting the nibble into the top four bits and arithmetic-shifting back down.

Verified identical to the original branching form (not merely to the intermediate one): over all 256 byte values, at every length around a vector boundary, and on a full-size 6.29M-value random expert, on both AVX2 and NEON.

Measurements

Standalone kernel, single-threaded: 11.84 → 0.18 ms on an i9-12900K, 3.19 → 0.08 ms on an Apple M4 Pro.

End to end — Qwen3.6-35B-A3B int4-gs64 (Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64), i9-12900K (avx-vnni, no AVX-512), Debian trixie, gcc 14.2, Samsung 990 Pro ext4, container fully in page cache, cap 32, N_NEW=48, MemoryMax=40G, three runs per rung, median:

SNAP=$M TOK=$M/tokenizer.json N_NEW=48 ./qwen36 32 4 prompt.txt

	tok/s	decode	TTFT
original (branchy scalar)	1.66	28.7 s	7.84 s
branchless scalar	2.60	18.5 s	5.21 s
vectorised	3.49	13.8 s	3.93 s

2.10x. The expert miss count is 6824 in every one of those runs, to the unit, and the hit rate never moves — placement behaviour is untouched, only what a miss costs. Generated text is character-identical on the real model, and the tiny fixture scores the same as before the change.

How it was located rather than guessed

Decode time on this model is a straight line in miss count — time = 11.1s + 2.62ms * misses predicts LRU, PILOT and pinned configurations within 1.1% — which put ~60% of decode inside the per-miss path and pointed here.

After the change the same fit reads 10.9s fixed + 0.43 ms/miss — fixed compute unchanged, as it must be for a patch touching only the unpack, and the per-miss term down 6.1x. The unpack is now 21% of decode instead of 62%, so the CPU bottleneck for this engine has moved to the matmul and attention path.

That the per-miss cost was CPU rather than I/O is confirmed most simply by this patch working: vectorising an integer unpack cannot produce 2.10x on a workload bound by storage. Independently, bracketing the runs against /proc/diskstats shows 9.49 GB of device reads per run — about 1.6 s at this drive's measured 5.78 GB/s, so ~6% of the original 28.4 s decode, and essentially accounted for by the one-time dense-weight load (~9.25 GB) rather than by expert misses.

Possibly of interest for #939 (very slow cache warmup), since warmup is all misses; and as a datapoint for #442, where the same class of win is blocked on float reassociation that does not apply here.

Checks

• make check on the 12900K: 704 tests, OK (skipped=35)
• Build warnings unchanged: 4 on this branch, the same 4 on pristine dev (all pre-existing, in the JSON emission path)
• Disassembly confirms the intended instructions on both gcc/AVX2 and clang/NEON

[OPS-NeoRetro]

@lornecodes, I felt something weird because I only see SSE4.1, SSE3 or SSE2 intrinsics in the AVX2 path.

[OPS-NeoRetro]

🤔 Why you didn't take advantage of 256 bits for AVX2 though? We need an SSE4.1/SSE4.2/SSSE3 path here.

[OPS-NeoRetro]

@JustVugg, merge #1232 first then take a look at this and #1239

[JustVugg]

Merging. I verified the bit-exactness claim independently rather than reading it: a standalone harness comparing your unpack_int4_to_int8 against the original branching form (not the intermediate branchless one) over all 256 byte values, every even length from 0 to 1024 so the vector boundary is crossed in both directions, and a full-size 6.49M-value random expert. Identical byte for byte, rerun on the current head after the rebase.

Kernel timing on my box, gcc -O2 -mavx2: 11.01 ms to 1.00 ms. Your machine's before-figure (11.84 ms) lines up with mine closely enough that the comparison is the same one.

Also checked: only qwen36.c is touched, so the blast radius is exactly one engine; token-exact at caps 1, 8 and 16; clean under a genuinely instrumented ASan build; the four qwen36 C tests pass; and the malformed-container refusal still fires.

The part I want on the record is how you found it. A fit of decode time against miss count, 11.1s + 2.62ms * misses predicting LRU, PILOT and pinned configurations within 1.1%, is a measurement that tells you where to look before you look. And re-fitting afterwards to 10.9s + 0.43ms/miss — fixed term unchanged, as it must be for a patch touching only the unpack — is the check most people skip.

The 9.49 GB of device reads from /proc/diskstats, accounted for by the one-time dense load rather than by expert misses, is what makes "this was CPU, not I/O" a demonstration instead of an assertion.

@OPS-NeoRetro's sequencing was right and has been followed: #1232 is merged, and #1239 has been reviewed — it is red on two defects in its own test scaffolding (a memcmp assertion that only holds when the compiler contracts FMA, and _sse41 targets that pass x86 flags on arm64), both reported there with the isolation.