[Feature]: Contribution scope for DeepSeek V4 MTP and opt-in approximate kernels

[DrewZt]

Problem

Now that #165 has landed, I would like to continue helping with DeepSeek V4 performance work without duplicating work already underway.

The current V4 performance branch appears to include verified drafting / MTP work. I also have other experimentally viable optimization directions, such as W4A8 execution paths, that may improve speed or memory use but can slightly change numerical results and may not remain token-exact at every near tie.

Before implementing another large change, I would like to confirm which direction the maintainers actually want.

Proposed solution

Could the maintainers clarify whether contributions are wanted in either of these areas?

1. MTP / DSpark: Is there any remaining part of the MTP or DSpark implementation, validation, or optimization that you would like me to own, or is the current performance work intended to cover it fully?
2. Opt-in approximate optimization: Would an experimental W4A8 or similar path be acceptable if the existing reference path remains unchanged and is still the default?

If an approximate path is welcome, I would keep it isolated behind an explicit opt-in flag and provide:

• the unchanged exact/reference fallback;
• wall-clock, tok/s, memory, and I/O comparisons;
• numerical-error and token-divergence measurements against the reference path;
• results across multiple representative prompts rather than a single favorable case;
• clear documentation that the mode trades a small amount of numerical fidelity for performance;
• portable fallback behavior on platforms without the optimized kernel.

I would first publish the experiment and measurements, then submit production integration only if the results justify it.

Alternatives considered

• Restrict future work to token-exact optimizations only. This is the safest policy, but excludes potentially useful speedups on constrained consumer hardware.
• Keep W4A8 as an external experiment until the accuracy/performance tradeoff is better understood.
• Wait for the current V4 performance work to land, then contribute only any missing validation or platform-specific kernels.

Scope and compatibility

The exact DeepSeek V4 target path would remain available and unchanged. Approximate kernels would be disabled by default and clearly identified in logs and documentation. Target-only / --no-dspark operation would continue to work independently.

Please also let me know the preferred base branch and validation bar for this kind of work?for example, whether numerical error and token-divergence measurements are sufficient, or whether you would also want perplexity or downstream task evaluation before considering a PR.

[JustVugg]

Asking before building is the right instinct, and the answer to both questions is yes — but the MTP one needs numbers before you spend a week on it.

1. MTP / DSpark — implemented, merged, and off

#839 landed all of it: the markov drafter, full three-stage DSpark, the attention snapshot/restore, and the accept rule. It is correct — a draft can save forward passes but never change a token, because every accepted token is still the target's own argmax, and rejection restores an exact snapshot and replays only verified inputs at unchanged absolute positions.

Then it was measured on real multi-turn chat, and it does not pay on this engine:

path	accepted	cost of the misses
prompt-lookup / markov	1 in 15	recurrent-state replay dominated the visible decode
full three-stage DSpark	10 in 24	one 14-token answer took 495 seconds

So V4_DRAFT and V4_MTP both default to 0, and the code stays with those numbers beside it in c/coli.

That is not a closed door — it is the most interesting open question in this engine, and it is unowned:

Why is acceptance that low, and is it the drafter or the verifier? A 1-in-15 markov hit rate on ordinary chat is worse than the DSpark paper implies, and nobody has separated "the drafter proposes badly" from "our verification window is too narrow". If the drafter is fine and the replay is what kills it, the fix is in the attention snapshot path, not in speculation at all.

And the structural reason it hurts here: rejection on this engine is expensive because the compressor carries recurrent kv_state/score_state rather than per-position rows, so a rejected suffix cannot be truncated — it must be replayed. On GLM, MLA rows are position-addressed and a rejection costs almost nothing. That asymmetry is worth understanding properly, and you know that attention path better than anyone.

Take that if you want it. It is measurement work first and code second, which is why nobody has done it.

2. Opt-in approximate kernels — yes, and W4A8 specifically

The project's stated guarantee is that the default never silently changes precision or router semantics. It says nothing against an approximate path the user asks for. Your framing already meets it: exact path unchanged and still default, approximate disabled by default, identified in logs and docs, portable fallback.

Two conditions, both from being burned before:

It must announce itself at runtime, not only in docs. #813 is open right now because a Metal backend printed "GPU-enabled" and silently ran everything on CPU — the counters that would have contradicted it were never printed on the path users actually run. An approximate mode that is quiet is the same failure with worse consequences.

"Opt-in" must mean the flag, not the build. If W4A8 is compiled in but reachable only through an env var, the exact path must stay bit-identical when the var is unset — and there should be a test that pins that, not just an intention.

3. The validation bar

Numerical error and token divergence are necessary and not sufficient. Perplexity alone is not enough either — it moves less than task accuracy on quantisation changes.

What has actually settled arguments in this repo is a task benchmark at n≥200 on identical hardware, back to back. #326 was closed on exactly that: hellaswag at n=200 gave grouped int4-g64 87.0% vs per-row 83.5% acc_norm, and the same comparison at n=40 had pointed the other way. Small-n is a smoke test, not evidence.

So: token divergence rate and numerical error for the mechanism, plus one task benchmark at n≥200 for the quality claim. Publish the experiment first as you proposed — that is the right order and I will read it before you write production code.

4. Base branch, and the thing we want most

Base is dev. main is release-only.

And since you asked what is wanted: DeepSeek V4 has no GPU path at all. No CUDA, no Metal, no Vulkan — I checked before answering. Every other engine has at least one tier. The engine is CPU-only by construction today, and on the reference box that means tokens per minute rather than per second.

A CUDA expert tier for V4 — even hot-experts-in-VRAM with the CPU keeping the tail, the way colibri.c and backend_cuda.cu already work together — would be worth more to users than any approximate kernel. It is also a bigger piece of work, so I am not asking you to drop W4A8 for it. But if you want the highest-value unowned thing in this engine, that is it, and #837 has someone asking for exactly that on Windows.

Thank you for #165, and for asking this instead of arriving with 3,000 lines.

[ZacharyZcR]

@DrewZt @JustVugg — flagging a collision before it costs anyone a week, since section 4 names the CUDA expert tier as the highest-value unowned thing here.

It is not unowned as of a few hours ago. #772 was rescoped today into exactly that, and the timing is why nobody noticed: when @JustVugg wrote this answer, #772 was still an engine-plus-kernels PR, it had gone CONFLICTING against #165, and GitHub had stopped running its CI entirely — a conflicted PR has no merge commit to build, so it showed zero checks and looked dead. It is MERGEABLE with 16/16 green now.

What is already written and validated

backend_cuda_dsv4*.{cu,h} — dense matmul, batched attention, routed MoE expert groups, plus optional vLLM / FlashInfer / DeepGEMM paths behind unsupported flags. Around 4,900 lines with tests and the oracles they were validated against.

They are self-contained on purpose: backend_cuda_dsv4.h includes only <stdint.h>, and the .cu only its own header and the CUDA runtime. Nothing reaches into an engine. DSV4_CUDA_OBJ is empty unless CUDA=1.

What is missing, and it is the half I should not write

Kernels are not a tier until something decides which experts live in VRAM and when. That is residency planning, prefetch and the expert store — @DrewZt's machinery from #165, and the reason @JustVugg called the split the right one on #772.

So the honest state is: the compute half exists and is tested; the placement half does not exist for V4. #772 alone still will not run DeepSeek V4 on a 3090, and I said so on the PR rather than let it look finished.

Suggested division, and I am flexible on all of it

• Kernels: mine, already done. Please do not rewrite them — if they are wrong or the wrong shape, tell me and I will change them.
• Placement and the call sites: yours. backend_cuda_dsv4.h's entry points are mine to fit to your expert store, not the other way round. If the shape you want differs from what it exposes, name it and I will change the header.
• W4A8 stays entirely yours and does not interact with any of this — it is a numerics question inside the kernels, and the exact path staying default is orthogonal to who wrote them.

That leaves you the part you know best and the part @JustVugg says is measurement-first, and it does not spend your week reproducing something that already passes its tests.

@Nekromanser — #837 is asking for the Windows CUDA build of this. The kernels being in place is the first half of the answer there too, though the engine-side wiring above still gates it.

One caveat on my own claim: the GPU tests need a device I do not have, so they are compile-checked only in CI. The two pure-CPU unit tests (dsv4-mhc, dsv4-quant) build and pass. If either of you has a card free, that is the one gap I cannot close myself.

[DrewZt]

@JustVugg @ZacharyZcR Thanks for the detailed feedback and for taking the time to coordinate the next steps here. Unfortunately, I need to close this issue due to an unexpected change in my employment responsibilities.

My company is now planning to have me take responsibility for private deployment-related work. That creates a potential overlap between my job responsibilities and further work in this area, especially around model deployment and GPU execution.

If I continue contributing to closely related work here, it could give my employer grounds to treat these contributions as part of my employment responsibilities, which could potentially put my stock options at risk. Given that, I need to maintain a clear boundary and stop working on this direction for now.

This is purely because of the change in my employment situation, not because I've lost interest in the technical work. I really appreciate the detailed guidance and the coordination around the V4 work, and I'm sorry that I have to step away just as the next steps were becoming clear.

Hopefully I can contribute again in areas that don't overlap with my work responsibilities in the future. For now, I'll close this issue.