[Feature]: METAL Attention support for fmt:4

[RDouglasSharp]

Problem

Currently line 2366 in colibri.c gates METAL support to fmt:2 only:
&& l->kv_b.fmt==2. The Metal mm_gemv kernel (line 16) only supports fmt=1 (int8), fmt=2 (int4 per-row), fmt=3 (int2), and f32 — no fmt=4 (grouped int4).

As a result, PROF=1 PROMPT="Compare the myths of Lucifer and Prometheus" NGEN=64 COLI_METAL=1 DIRECT=1 MTP=0 TEMP=0 COLI_METAL_RESSET=1 SNAP=~/glm52_i4 ./colibri 64 achieves 2.39 tok/s on fmt:2 but only 0.81 tok/s on fmt:4

Proposed solution

Add fmt:4 support to the Metal attention path. This involves

• Adding gs to AttnW struct and bind_gemv
• Adding grouped-int4 handling in the mm_gemv Metal kernel
• Relaxing the l->kv_b.fmt==2 guard

Alternatives considered

We could leave processing as is, but the existing code performs Apple Silicon Mac attention processing on the CPU. This is ~3X slower than the fmt:2 model on my machine.

Scope and compatibility

Limited to fmt:4 on Mac apple silicon

[RDouglasSharp]

Profile output for the two formats:

PROF=1 PROMPT="Compare the myths of Lucifer and Prometheus" NGEN=64 COLI_METAL=1 DIRECT=1 MTP=0 TEMP=0 COLI_METAL_RESSET=1 SNAP=~/glm52_i4_g128 ./colibri 64
prefill 9 tokens in 10.64s | decode 64 tokens in 79.43s (0.81 tok/s) | expert hit rate 72.7% (pin 57.6% + lru 15.1%) | RSS 95.27 GB
experts loaded/token: 590.6 (per-layer 7.88 across 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculation: 1.02 tokens/forward (63 forwards per 64 tokens) | MTP acceptance 0% (0/0)
PROFILE: expert-disk 58.157s service / 18.979s wait | expert-matmul 7.481s | attention 48.441s (including kvb 0.000s) | lm_head 0.000s | other 4.527s
ATTENTION: projection/RoPE 15.995s | score-softmax-value 4.806s | output projection 27.634s
P0-EXEC: routed CPU 0.000s / 0.00 GB/s (0 row) | routed GPU critical 0.000s | router 0.990s | residual P2P 0.000s / 0 hop | orchestration 3.537s
METAL: blocchi GPU 8990 | fallback CPU 0 | expert su GPU 46179 | setup 1.13s gpu-wall 6.47s (kernel 9.39s) scatter 0.14s
METAL-RESSET: flush 0.51s (residency-set commit in moe_submit, outside setup/gpu-wall)
[PROF] decode forwards: 63 | latency p50 1257.8 ms | p90 1301.6 ms | p99 1325.4 ms | max 1364.1 ms | 1.02 tok/forward
[PROF] expert I/O: 207.278 GB fetched (3238.7 MB/token, 2.61 GB/s over the run) | hit 72.7% (21766 pin + 5698 lru / 10336 load) | 590.6 loads/token | 58.2s read service / 19.0s felt wait
[PROF] DISK-CLASS (recency split): cold 10262 x 205.79 GB @ 3.58 GB/s-thread, wall 18.8s @ 10.92 GB/s-wall, avg-conc 3.1 (direct 10226/10262) | warm 74 x 1.48 GB @ 2.73 GB/s-thread, wall 0.4s @ 3.60 GB/s-wall, avg-conc 1.3 (direct 74/74) | disk-busy 18.9s (24% of window)
[PROF] resident experts: 1961 pinned (39.3 GB) + 2550 in LRU (51.1 GB, cap 34/layer)
[PROF] time shares: expert-I/O 24% | expert-matmul 9% | attention 61% | lm_head 0% | other 4%
[PROF] P0 execution: routed CPU 0.000s / 0.00 GB/s (0 row) | routed GPU critical 0.000s | tier straggler 0.00x | router 0.990s | residual P2P 0.000s (0 hop, 0.000 ms/hop) | orchestration 3.537s
[PROF] verdict: attention-bound (61%) — context length is the cost (DSA not available for this model). A lower CTX helps if the workload allows.

PROF=1 PROMPT="Compare the myths of Lucifer and Prometheus" NGEN=64 COLI_METAL=1 DIRECT=1 MTP=0 TEMP=0 COLI_METAL_RESSET=1 SNAP=~/glm52_i4 ./colibri 64
prefill 9 tokens in 5.45s | decode 64 tokens in 26.82s (2.39 tok/s) | expert hit rate 71.2% (pin 48.4% + lru 22.9%) | RSS 92.72 GB
experts loaded/token: 590.6 (per-layer 7.88 across 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculation: 1.02 tokens/forward (63 forwards per 64 tokens) | MTP acceptance 0% (0/0)
PROFILE: expert-disk 55.849s service / 17.655s wait | expert-matmul 5.424s | attention 3.168s (including kvb 0.000s) | lm_head 0.000s | other 0.575s
ATTENTION: projection/RoPE 0.000s | score-softmax-value 0.000s | output projection 0.000s
P0-EXEC: routed CPU 0.000s / 0.00 GB/s (0 row) | routed GPU critical 0.000s | router 0.005s | residual P2P 0.000s / 0 hop | orchestration 0.570s
METAL-ATTN: layer GPU 4914 | gpu-wall 2.94s (kernel 2.01s | cpu-sched 0.05s gpu-sched 0.53s)
METAL: blocchi GPU 9046 | fallback CPU 0 | expert su GPU 41458 | setup 1.18s gpu-wall 4.48s (kernel 3.65s) scatter 0.11s
METAL-RESSET: flush 0.38s (residency-set commit in moe_submit, outside setup/gpu-wall)
[PROF] decode forwards: 63 | latency p50 428.9 ms | p90 501.8 ms | p99 524.9 ms | max 535.1 ms | 1.02 tok/forward
[PROF] expert I/O: 205.723 GB fetched (3214.4 MB/token, 7.67 GB/s over the run) | hit 71.2% (18284 pin + 8640 lru / 10876 load) | 590.6 loads/token | 55.8s read service / 17.7s felt wait
[PROF] DISK-CLASS (recency split): cold 10664 x 201.71 GB @ 3.71 GB/s-thread, wall 17.4s @ 11.57 GB/s-wall, avg-conc 3.1 (direct 10619/10664) | warm 212 x 4.01 GB @ 2.96 GB/s-thread, wall 1.0s @ 4.03 GB/s-wall, avg-conc 1.4 (direct 211/212) | disk-busy 17.5s (65% of window)
[PROF] resident experts: 2341 pinned (44.3 GB) + 2325 in LRU (44.0 GB, cap 31/layer)
[PROF] time shares: expert-I/O 66% | expert-matmul 20% | attention 12% | lm_head 0% | other 2%
[PROF] P0 execution: routed CPU 0.000s / 0.00 GB/s (0 row) | routed GPU critical 0.000s | tier straggler 0.00x | router 0.005s | residual P2P 0.000s (0 hop, 0.000 ms/hop) | orchestration 0.570s
[PROF] verdict: I/O-bound — 66% of the time waits on expert reads (hit 71%). More cache is the lever: raise RAM_GB (or add RAM). Try PIPE=1 (overlap reads with matmul).

[JustVugg]

Confirmed and actionable. Your profile makes the cost concrete: on the same prompt/machine, fmt=4 runs 0.81 tok/s vs 2.39 tok/s on fmt=2 — and the split shows why: attention is 61% of the time on fmt=4 (attention-bound) vs 12% on fmt=2. The fmt=2-only guard at the kv_b.fmt==2 gate is forcing the fmt=4 attention path off the Metal kernel.

Your PR #587 (Metal fmt=4 grouped-int4 decode: attention + routed experts) implements exactly the fix proposed here — it's MERGEABLE/CLEAN and CI-green. Two things before it lands:

1. Overlap with metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457 (metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes, @monotophic). Both add fmt=4 to the Metal path — metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457 on the mm_gemv kernel, Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 on attention + routed experts. If they touch the same kernel/bind code they'll conflict, and we don't want two implementations of the same fmt=4 decode. Could you and @monotophic reconcile — does Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 supersede metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457, or are they complementary (GEMV vs attention) and mergeable in sequence? Whichever lands first, the other should rebase on top.
2. Hardware confirmation. CI builds the Metal target but doesn't execute it on a GPU, so the green check proves it compiles, not that it's token-correct on Metal. You're running real Apple Silicon with the g128 container — can you confirm Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 produces token-correct output (matches the CPU path) on fmt=4, not just faster? Your before/after tok/s already shows the speedup; a correctness spot-check closes it.

Sort those two out and #587 is the merge. This is a solid, well-scoped win for Apple Silicon users on grouped-int4.

[monotophic]

We are working on the reconciliation now (@RDouglasSharp you can dm if more is needed than these thread comments). We have been sitting on some additional work that overlaps with PR #587 and are reconciling that as well. It's all great work.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and branches).

Reconciliation plan for #457 / #587 (concrete recommendations follow once the work below completes):

Provisional answer to the question: complementary, not superseding — #457 covers the dense GEMV kernel plus two latent fmt=4 fixes; #587 covers fused-attention kv_b and the routed-expert path. They collide only where #587 re-derives #457 content (it independently arrived at the same qt_from_disk scale-allocation fix — good sign for both). Sequence per @RDouglasSharp's own proposal: #457 lands first, #587 rebases on top, dropping its #457-twin hunks.

In flight now:

A hunk-by-hunk rebase map of #587 against #457 (drop-as-duplicate / keep / adapt, per hunk) — we'll hand it over so the rebase is mechanical rather than archaeological.
Independent validation of #587's current head on my Apple Silicon: build is clean and the full Metal test suite passes under both residency-set states; real-container checks on my g64 container (complementing the g128 evidence) are running now — this addresses the hardware-confirmation ask from a second machine and second group size.
Two small code-level findings from our review to fold into the rebase — we'll post details on #587.
Our own staged work, resolved: my private routed-experts branch (staged behind #457) retires in favor of #587's expert path — we're putting review and validation weight behind #587 instead. My pending kv_b fallback-notice PR is complementary to #587's 2a (diagnostic vs cure) and will be adapted to the new gate semantics before submission.

Follow-up with the map, the validation results, and the findings shortly.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and branches).

Follow-up: concrete reconciliation + validation results (completing the plan posted above).

1. Reconciliation recommendation

#457 first, #587 rebases on top — confirmed mechanical by construction: we built the integrated tree locally and it passes the combined test suite (ours + #587's new fmt=4 cases) under both residency states. The rebase reduces to:

• Drop (already in metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457, verified at source): the qt_from_disk scale-allocation fix (identical), the matmul_qt_ex fmt=4 gate, the mm_gemv fmt=4 kernel branch, and the bind_gemv gs plumbing.
• Keep (additive, Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587's real payload): a_deqrow/a_qabs/a_ctx fmt/gs awareness + both kv_b guard relaxations; the moe_gemv fmt=4 branch + qgs threading through moe_submit/coli_metal_moe_block(_begin) + the MB_BUILD group-size capture; all new tests.

We prepared a hunk-by-hunk disposition map (every #587 hunk: drop/keep/adapt, with citations) — @RDouglasSharp, happy to share it however is easiest.

2. Hardware validation on my M5 Max, g64 container (complementing the g128 evidence)

• Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587's head builds clean (zero warnings); full Metal test suite passes both COLI_METAL_RESSET states, including the new fmt=4 attention/moe oracle cases.
• Routed-expert path (g64, real weights): CPU↔Metal token-identical over a temp-0 probe generation, with the GPU path fully engaged (22k experts on GPU, fallback CPU 0).
• Attention path (kv_b at fmt=4, real weights): Metal output is byte-identical across repeated runs (deterministic), and CPU↔Metal diverge at a single early near-tie token with both continuations fully coherent. This is the engine-level cross-backend float-ordering class we established with a baseline control during metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457's validation (cross-backend temp-0 identity does not hold for any container/format in general on this engine; genuine decode corruption looks nothing like a clean one-token fork). By the bar metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457 was held to — kernel-level oracles + coherent generations + engaged-GPU counters — Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 is token-correct on fmt=4.

3. Two findings to fold into the rebase

1. The matmul_qt_ex gate relaxation is inert as submitted: coli_metal_gemm's internal format guard still rejects fmt=4 on current dev (clean CPU fallback — layered guards doing their job). It becomes live through metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457, which is one more reason the sequencing is right; it drops with the other twins.
2. MB_BUILD captures one group size per GPU batch, and the shared-expert inclusion checks format equality but not gs equality — a future container mixing group sizes would silently decode the shared expert with the wrong gs. No container that exists today triggers it; we have a small tested guard (defers mismatched subsets to the existing CPU fallback) ready to contribute during the rebase if wanted.

Net: two complementary PRs, one mechanical rebase, both validated on real Apple-Silicon hardware across two group sizes. Ready when you are.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and containers).

Correction to our validation comment above — one claim was mis-attributed, caught tonight by an instrument we had in the integrated tree.

Our "attention path (kv_b at fmt=4)" arm did NOT exercise #587's fused-attention change. My g64 container's kv_b is int8 (fmt=1) without its overlay — resident-class tensor, 8-bit recipe — so the fused path was gate-rejected in both backends of that comparison, and what we actually validated there was the routed-experts path under CPU-attention fallback. The claims that stand unchanged: build/test-suite results, and the g64 routed-experts token-identity (that arm's GPU engagement is counter-verified). The 2a attention evidence currently rests on @RDouglasSharp's own g128 container, which genuinely has grouped kv_b and is the right substrate for it.

Two constructive outcomes:

1. How we caught it: the fallback-notice patch we have staged (the kv_b diagnostic PR mentioned above, adapted to the post-Metal fmt=4 grouped-int4 decode: attention + routed experts (#585) #587 gate) fired on the run and named the actual format: kv_b_proj is fmt=1 (not int4) on 78/78 layers ... requantize kv_b to int4 (per-row or grouped). Silent CPU fallback is exactly how this class of mis-attribution happens — we mis-attributed with the counters in hand; without the notice it would have survived longer.
2. Completing the g64 datapoint properly: no container we own had fmt=4 kv_b, so we minted a grouped-int4 (g64) kv_b overlay (via the same converter-derived quantization path, layout-verified) and the CPU-vs-Metal comparison is re-running now with the fused fmt=4 path counter-verified engaged — against @RDouglasSharp's merged-457-587 branch, which builds clean and passes the Metal suite on our machine (and carries the metal: grouped-int4 (fmt=4) GEMV support + two latent fmt=4 fixes #457 commits with authorship intact — appreciated). Results follow here shortly; one note for the merge plan: that branch's base is dev f1b0f7e (pre-Inkling), so the current-dev rebase remains the one open step either way.

Apologies for the noise — better corrected than compounding.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and containers).

2a validation complete — and one important finding on the temporary merge branch.

The real fmt=4 attention validation (g64)

We minted a grouped-int4 (g64) kv_b overlay for my container (converter-derived quantization, layout-verified) and ran the comparison on a #457+#587 integration on current dev, with engagement counter-verified this time (METAL-ATTN present; our fallback diagnostic silent):

• CPU ↔ Metal: token-identical over the temp-0 probe, fused fmt=4 attention path engaged.
• Metal ↔ Metal: byte-identical across repeated runs (deterministic).

Combined with the earlier g64 routed-experts identity and @RDouglasSharp's g128 results, both halves of #587 now have real-weights validation on both group sizes. On the bar set for #457 — kernel oracles + real-container behavior + engaged-GPU counters — #587 is token-correct on fmt=4, attention included.

Finding on merged-457-587 (@RDouglasSharp — two-line fix)

The temporary merge branch builds clean and passes the Metal suite, but the colibri.c conflict resolution dropped the two guard relaxations that make 2a reachable: lines 2353 and 4078 still read l->kv_b.fmt==2 (the kernels and one relaxed size expression at 1083 made it through — partial application). As merged, fmt=4 kv_b silently falls back to CPU attention and the branch ships the experts path only. Fix is the two one-liners: l->kv_b.fmt==2 → (l->kv_b.fmt==2||l->kv_b.fmt==4) at both sites. (Also noting the branch base is dev f1b0f7e, pre-Inkling — the current-dev rebase remains the open step, as planned.)

This is precisely the conflict-archaeology class we built the hunk-by-hunk disposition map for — attached/available, covering every #587 hunk against #457 (drop-as-duplicate / keep / adapt, with citations), plus the tested shared-expert group-size guard from our review, both ready to fold into the rebase.

The systemic note

Today's tally of silent CPU fallback incidents across our two teams: a configuration mis-attribution (ours, corrected above), and a merge that quietly locked out a validated feature (this one) — both invisible to green builds and passing suites, both caught only by counters or by the one-line fallback notice we have staged as a follow-up PR. The case for loud fallbacks writes itself; we'll submit that notice once this queue clears.

Sequencing unchanged and everything needed is on the table: #457 (ready on current dev) → #587 rebased with the map — mechanical from here.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and containers).

The full hunk-disposition map, as flagged in the 2a-results comment above — attached.

Every #587 hunk (head 2219144, merge base 11cd1c6) disposed against #457 (head 649b552): DROP-as-duplicate / KEEP-VERBATIM / ADAPTED, each with citations into both branches. It covers the two colibri.c gate relaxations whose drop on the temporary merge branch we flagged above, and the shared-expert group-size guard from our review rides along as a noted non-#587 extra.

Totals: 30 hunks — 15 keep-verbatim, 8 adapted, 7 drop (5 true twins already on #457: the mm_gemv fmt=4 branch, bind_gemv gs plumbing, the matmul_qt_ex gate, the falloc→qalloc fix, and the t_mkw_g test helper).

Coordinates are in the file's header; if either head moves before the rebase, the dispositions hold and (we) will refresh the hunk numbering on request.

@RDouglasSharp — this plus the two-line fix should make the post-#457 rebase mechanical.

REBASE_MAP_587_vs_457.md

[JustVugg]

Implemented. The gate this issue named (&& l->kv_b.fmt==2) is now l->kv_b.fmt==2 || (l->kv_b.fmt==4 && !g_moe_exact) (colibri.c:3898), the byte-size computation handles fmt=4 alongside fmt=2 (:2091), and an mb_gs_compat() helper checks group-size compatibility for the fused path. Metal now runs grouped int4 (fmt=4) MoE experts and fused attention, plus fmt=5/6/8 and kv_b grid chunking that came with it.

Worth noting the history: @aaristov's #829 originally implemented this, dev then implemented it independently and went further, and he resolved all 37 conflicting hunks to dev's side rather than fight it, leaving only the observability fixes that merged in v1.7.0. Both routes led here.