From e2c3d941c1934b2e02474d8780ede67b1bcff770 Mon Sep 17 00:00:00 2001 From: Steve Markgraf Date: Mon, 27 Jul 2026 18:49:11 +0200 Subject: [PATCH] =?UTF-8?q?quant:=20AVX2=20matmul=5Fe8=20=E2=80=94=20fmt?= =?UTF-8?q?=3D6=20was=2092%=20of=20decode=20on=20a=20scalar=20kernel?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fmt=6 shipped with only its scalar reference kernel, and on a real model that dominated everything else: on GLM-5.2 (744B, E8 container) a 32-token decode spent 206.6s of 224.1s — 92% — inside matmul_e8, with disk service fully overlapped (1.1s of actual wait). The format's own arithmetic was the wall, not I/O and not the GPU. This is the same shape of problem AVX2 matmul_i3 fixed for int3-g64, where the scalar path cost more than half the achievable rate. The format suggests the vectorisation: one 8-weight lane is two 4-dim grid rows (8 contiguous codebook bytes) plus 8 signs, which is exactly one AVX2 register. So a lane decodes straight into a register and FMAs against x, instead of being expanded into a stack buffer and read back: - the 8 grid bytes widen with a single vpmovzxbd; - the sign byte (the 7 stored bits plus the parity-derived 8th) expands to 8 lane masks with AND/CMPEQ against a bit-select vector and applies as an XOR of the float sign bit. That removes the per-weight branch, which is what made the scalar expansion expensive; - the grid's half-unit convention folds into the sub-scale, so the magnitudes need no separate scaling; - two accumulators over a super-block's 32 FMAs keep it off one dependency chain, and one horizontal add per 256 weights replaces one per 32. The scalar path stays for the ragged tail and for non-AVX2 hosts; the converter enforces whole 256-blocks, so the fast path is the one real containers take. Kernel: 280.8 -> 4141.3 Mw/s single-threaded, 14.7x (44.8 -> 3.0 ms per [2048,6144] expert tensor). tests/test_e8_kernel green against its fixture (worst rel 1.46e-06); AVX2 vs scalar agree to 2.2e-07 relative L2, which is accumulation order. End to end on GLM-5.2 (RX 9070 box, 32-token greedy decode, cold cache, single drive), same text produced in every configuration: prefill decode expert-matmul CPU before 91.3s 224.1s (0.14 tok/s) 206.6s CPU after 16.7s 38.6s (0.83 tok/s) 18.7s VK before 91.6s 177.3s (0.18 tok/s) 161.0s VK after 17.2s 33.6s (0.95 tok/s) 16.8s 5.3-5.9x decode, 5.5x prefill. Decode is now disk-bound again (34.9s service, 3.4s wait) rather than compute-bound, which is the regime the streaming design targets, and the VK tier's share rose from 22.7% to 31.0% of expert hits. --- c/quant.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/c/quant.h b/c/quant.h index 505141f5a..70fcf5c72 100644 --- a/c/quant.h +++ b/c/quant.h @@ -1200,6 +1200,52 @@ static void matmul_e8(float *y, const float *x, const uint8_t *q, const float *u uint16_t dh; memcpy(&dh, blk+96, 2); float d=e8_fp16_to_f32(dh); int base=(int)(b*E8_QK); +#ifdef __AVX2__ + /* One 8-weight lane is exactly one AVX2 register, which is what the + * format's own shape suggests: a lane is two 4-dim grid rows (8 + * contiguous codebook bytes) plus 8 signs. So instead of expanding a + * sub-block into a stack buffer and re-reading it, each lane is + * decoded straight into a register and FMA'd against x: + * - the 8 grid bytes widen with one vpmovzxbd, + * - the sign byte (7 stored bits + the parity-derived 8th) expands + * to 8 lane masks with an AND/CMPEQ against the bit-select vector + * and is applied as an XOR of the float sign bit — no branches, + * which is what made the scalar expansion expensive, + * - 0.5 (the grid's half-unit convention) folds into the sub-scale. + * Two accumulators over the 32 FMAs of a super-block keep this off a + * single dependency chain, and one horizontal add per 256 weights + * replaces one per 32. */ + if(base+E8_QK<=I){ + const __m256i sel=_mm256_setr_epi32(1,2,4,8,16,32,64,128); + const __m256i sgn=_mm256_set1_epi32((int)0x80000000u); + __m256 ac[2]={_mm256_setzero_ps(),_mm256_setzero_ps()}; + for(int ib=0; ib>28)&0xF))*0.5f; + __m256 vdb=_mm256_set1_ps(0.5f*db); + const uint8_t *ix=blk+ib*8; + int off=base+ib*E8_SUB; + for(int l=0;l<4;l++){ + uint32_t sv=(word>>(7*l))&0x7Fu; + uint32_t s8=sv|((uint32_t)__builtin_parity(sv)<<7); /* odd parity closes the lane */ + uint32_t g0,g1; + memcpy(&g0,e8_grid[ix[l*2+0]],4); + memcpy(&g1,e8_grid[ix[l*2+1]],4); + __m128i by=_mm_cvtsi64_si128((long long)((uint64_t)g0|((uint64_t)g1<<32))); + __m256 v=_mm256_mul_ps(_mm256_cvtepi32_ps(_mm256_cvtepu8_epi32(by)),vdb); + __m256i m=_mm256_cmpeq_epi32(_mm256_and_si256(_mm256_set1_epi32((int)s8),sel),sel); + v=_mm256_xor_ps(v,_mm256_castsi256_ps(_mm256_and_si256(m,sgn))); + ac[l&1]=_mm256_fmadd_ps(v,_mm256_loadu_ps(xs+off+l*8),ac[l&1]); + } + } + __m256 t=_mm256_add_ps(ac[0],ac[1]); + __m128 h=_mm_add_ps(_mm256_castps256_ps128(t),_mm256_extractf128_ps(t,1)); + h=_mm_add_ps(h,_mm_movehl_ps(h,h)); + h=_mm_add_ss(h,_mm_shuffle_ps(h,h,1)); + acc+=_mm_cvtss_f32(h); + continue; + } +#endif for(int ib=0; ib=I) break;