[Feature]: Metal backend for kimi_k3 — K3's GPU path is Vulkan-only, so Apple Silicon falls back to CPU

[Toao27]

Problem

Kimi K3 is the only tier-1 model with no usable GPU path on macOS, which puts
it out of reach on Apple Silicon regardless of storage.

kimi_k3.c has no Metal support at all:

$ grep -c COLI_METAL kimi_k3.c
0

The Makefile confirms this is by design rather than a build quirk on my end
(Makefile:715):

# kimi_k3.c contains no COLI_CUDA code at all — its GPU path is Vulkan. But

The generic rule still accepts and passes through the Metal flags, so the build
looks like it succeeded when it hasn't:

$ make kimi_k3 METAL=1 ARCH=native
clang ... -mcpu=native -DCOLI_METAL kimi_k3.c   -o kimi_k3 ... -framework Metal -framework Foundation -lc++
kimi_k3.c:255:12: warning: unused variable 'g_k3_vk' [-Wunused-variable]
  255 | static int g_k3_vk=0;     /* backend live (K3_VK=0 disables) */

Two things there: backend_metal.o is absent from the link line — compare the
working GLM build, which includes it —

clang ... -DCOLI_METAL colibri.c  backend_metal.o  -o colibri ... -framework Metal

— and g_k3_vk is unused, so the Vulkan backend didn't compile in either. The
resulting binary is CPU-only with Metal frameworks linked but nothing behind them.

MEASURED IMPACT

I benchmarked Metal vs CPU on GLM-5.2 on an M1 Max (64 GB, 32-core GPU,
commit df2c248) in #1030

RAM_GB=24 Metal 0.49 / CPU 0.20 tok/s = 2.45x
RAM_GB=52 Metal 0.33 / CPU 0.13 tok/s = 2.54x

Prefill is more lopsided still: ~31 s Metal vs ~111 s CPU (3.6x), consistent
with docs/metal.md noting that decode on Apple Silicon is matmul-bound.

Tuned, that machine reaches 0.74 tok/s on GLM-5.2. Without Metal it sits near
0.13 — the difference between usable and not.

For K3 the CPU-only penalty compounds: it routes 16 of 896 experts per layer
against GLM's 8 of 256 (roughly twice the expert bytes per token); at ~1.6 TB
the container won't fit most laptops' internal storage, so it's read over
Thunderbolt (~2.8 GB/s) rather than internal NVMe (4.4 GB/s measured here);
and residency falls accordingly — I see 13% on GLM-5.2 at RAM_GB=24, so K3
would be low single digits.

Extrapolating from the CPU figures, that lands near 0.05 tok/s. A 60-token
answer would take roughly 20 minutes.

Proposed solution

Three options, in rough order of effort — any one of them would help.

1. A native Metal backend for kimi_k3.c, reusing backend_metal.mm as
   colibri.c does. The kernels GLM needs — batched routed-expert SwiGLU,
   fused decode attention, prefill GEMMs — look like they map onto K3's
   shapes, though K3's larger expert count and different attention may
   need work I can't judge from outside the source.

2. Failing that, fail loudly. make kimi_k3 METAL=1 currently succeeds and
   produces a binary with no GPU path. A build-time warning, or a startup
   line reading "no GPU backend on this platform", would save people the
   discovery I just made — ideally before they commit to a 1.6 TB download.

3. Document the per-engine backend matrix, e.g.

   engine	CUDA	Vulkan	Metal
   colibri (GLM-5.2)	yes	—	yes
   kimi_k3	no	yes	no
   deepseek_v4	?	?	?
   olmoe	?	?	?

   docs/metal.md describes the Metal backend without saying which engines it
   covers; I assumed it was engine-agnostic until I read the source.

Point 3 alone would have answered my question in seconds and costs almost
nothing.

Alternatives considered

• Vulkan via MoltenVK. macOS has no native Vulkan driver, so this needs
  MoltenVK translating to Metal underneath plus a working SPIR-V toolchain
  for the VK_SPV shaders. Even if wired up, it adds a translation layer on
  top of the API a Metal path would target directly. A native backend looks
  like the better investment.

• Accepting CPU-only. At ~0.05 tok/s this isn't a slow-but-usable
  configuration; it's roughly 20 minutes for a short answer, on top of
  €250 of external storage and a 1.6 TB download.

• Staying on GLM-5.2. This is what I'm doing, and it's a reasonable answer —
  GLM sits within a few index points of K3 on published benchmarks, ships
  under MIT rather than a bespoke licence, and has a working Metal path. But
  it means K3 is effectively unavailable to Apple Silicon users rather than
  merely slow, which seems worth naming.

Scope and compatibility

• Zero-dependency CPU path: unaffected. This is purely additive, gated behind
  METAL=1, exactly as the GLM Metal path is today. A build without METAL=1
  produces the same binary it does now.
• CUDA path: unaffected. kimi_k3.c contains no COLI_CUDA code (Makefile:715),
  so there's no interaction.
• Vulkan path: unaffected — this would be an alternative backend, not a
  replacement. Presumably the same runtime gating that K3_VK uses.
• Model formats: no change. K3 streams MXFP4 experts natively; this is about
  where the arithmetic runs, not how weights are stored.
• Platforms: macOS / Apple Silicon only. No effect on Linux, Windows or PowerPC.
• Options 2 and 3 have no runtime effect at all — a build warning and a docs
  table.

Environment for the observations above: MacBook Pro, M1 Max (10 CPU / 32 GPU
cores), 64 GiB unified memory, macOS 26.6.1 (25G76), Apple clang 21.0.0,
Homebrew libomp 22.1.8, commit df2c248
(v1.6.1, tip of main). The GLM-5.2 Metal build works correctly on this machine.

[JustVugg]

Your diagnosis is correct and the Makefile comment you found says it outright: Kimi K3's GPU path is Vulkan, and there is no Metal code in kimi_k3.c at all. On Apple Silicon it falls back to CPU. That is a real gap, not a build quirk on your side — and the fact that the generic rule still accepts the Metal flags without doing anything is its own small wrong, because it makes the build look like it succeeded at something it never attempted.

Good news: someone is already on it. #790 (Metal (Apple GPU) backend for Kimi K3, +2,840 lines) implements exactly this. It has been open for two weeks and currently conflicts with dev — so it needs a rebase before it can be reviewed, not a rewrite.

If you have Apple Silicon and are motivated, the most useful things right now are, in order:

1. Test Metal (Apple GPU) backend for Kimi K3 #790 on your machine once it's rebased and post numbers — Metal PRs in this repo live or die on someone with the actual hardware confirming both correctness and speed.
2. Report which K3 container format you'd be using (int4-g64 / int8), because the Metal decode path is format-sensitive — see fix(serve/coli): make GPU-vs-fallback counters and chat status actually visible #829 and engine: notice when a tensor's format silently disables the fused Metal decode path #827 for how that has bitten the Metal backend before.

Keeping this open as the tracking issue for the capability, and linking #790 as the implementation. Labelled metal + model-support.

[Toao27]

Happy to be the Apple Silicon tester for #790, with one honest caveat: I don't currently have a K3 container. My GLM-5.2 work was on the internal SSD, and K3's ~1.6 TB doesn't fit — I'll need a Thunderbolt enclosure, so I'd be a slower tester than for GLM.

Which raises a question that might unblock a lot of this: does #790 ship a standalone kernel test, the way make metal-test validates the GLM backend against a CPU reference with no model present? If so I can verify correctness on an M1 Max immediately, today, and that's the half of the review that doesn't need 1.6 TB. Only the throughput number needs the container — and knowing correctness holds first would make the hardware spend a lot easier to justify.

On format: no preference yet, since I'd be starting fresh. Given #827 and #829, I'd rather take whichever format the Metal decode path handles best — happy to be told which to fetch.

Baseline for comparison, measured on this machine (M1 Max, 64 GB, 32-core GPU, df2c248) on GLM-5.2: Metal 2.45–2.54× over CPU across --ram settings, 3.6× on prefill, 0.74 tok/s tuned. Details in #1030. Worth noting from that work: sequential runs on this laptop thermally throttle badly, and gpu-sched tracks it — anyone benchmarking #790 on Apple hardware should watch that number or they'll measure their heatsink.

[JustVugg]

Implemented in v1.7.0 by #1113 (originally #790, @RDouglasSharp): Kimi K3 now has a Metal backend, aligned KDA state and window buffers, a wrap-once buffer cache and a CPU-side MLA KV cache. Measured 1.7x/2.4x on the compute-bound phases (KDA attention and projections dispatched to the GPU); MoE experts stay on the CPU. K3's first GPU backend, and the largest model we support now runs on Apple Silicon.