Run Qwen3.6-35B-A3B MoE on GTX 1080 Ti

[SeanPedersen]

Does this project allow me to run Qwen3.6-35B-A3B Q4_K_XL faster than llama.cpp on my GTX 1080 Ti (11GB VRAM) because of MoE aware GPU caching?

I am looking for this optimization:

llama.cpp partitions a MoE model vertically, layer by layer, across CPU RAM and GPU VRAM. As a result, no token is processed entirely on the GPU, placing the CPU on the critical path for every token and creating a substantial performance bottleneck.

The idea: if a small subset of experts of the whole network is only active for most token, we should slice vertically so that the hot (active) experts are cached on GPU and cold ones on CPU. This would allow most tokens to be processed solely on the GPU, leading to massive speedups.

asking because of this post: https://www.lucebox.com/blog/spark

[davide221]

Hi @SeanPedersen yes, I think none tried our engine with this card. But the premise is that it should be faster based on the expert calibration with Luce Spark

[SeanPedersen]

Tested on a GTX 1080 Ti (11GB VRAM) with Qwen3.6-35B-A3B Q4_K_XL.

Luce Spark works and, with a 10 GiB budget, improved short decode from ~30.8 to ~35 tok/s, slightly faster than my llama.cpp baseline at 32.5 tok/s. However, at 128K context with Q8 KV, a 108K token prefill was substantially slower than llama.cpp and did not finish within the baseline’s 486-second total runtime, so Spark did not win overall.

So promising short-decode gains, but llama.cpp remains faster overall for long-context workloads on this system (i9 18 core CPU, 64GB RAM + GTX 1080 Ti).

[davide221]

@SeanPedersen try to use it with Luce PFlash, it will cut dramatically the prefill cost