Conversation
…-ID intake SUBMIT grows a key=value extension namespace past the 7th field (logprobs=k opts a request into per-token numeric emission, capped at run_ablate_score's top-32 ceiling; ids=1 marks the payload as ASCII token ids parsed straight into the prompt buffer, bypassing tok_encode with no detokenize/re-encode round trip). Old engines reject any extended header: both legacy sscanf arms fail on the extra field and answer ERROR 0 BAD_REQUEST. The rejection itself is safe, but the reject path does not drain the rejected request's payload bytes, so the engine re-reads them as protocol lines -- the pre-existing behavior of every malformed-SUBMIT reject, not new exposure added by the extension namespace. Opted-in requests emit one ECHO frame per echoed prompt position (log-softmax + unsorted top-k table, position 0 carries nan: nothing to condition on) from a new live prefill read-out loop structurally parallel to run_score's, and their DATA frames carry the same numeric tail. spec_decode's emit callback now hands each token its scoring logit row, so accepted draft tokens have no numeric gap. Requests that do not opt in keep byte-identical frames and step()'s single last-position lm_head cost. The server dispatcher accepts ECHO and extended DATA without raising (a new frame reaching an old dispatcher kills every in-flight request, which is why this engine half lands first and stays inert until a request opts in); the numeric fields are consumed by the server feature half.
…e audit golden_fixture_capture.py captures a battery of non-logprobs HTTP responses (chat, tools, streaming, plain completions, the error cases that must not move, /v1/models) against a running server, normalizes ids/timestamps, and byte-diffs a pre-change capture against a post-change one -- the mechanical form of the 'everything that worked before still works' invariant, built before the opt-in channel lands so it is the proof rather than reviewer inspection. check_data_logprob_gaps.py audits a raw engine stdout transcript of an opted-in run: every generated DATA frame must carry the numeric tail (accepted draft tokens bypass the mux pick_tok sites, so a speculative run is where a gap would hide), ECHO positions must be contiguous from 0, and position 0 must carry nan.
…n fields
The extended-header arm accepted "512logprobs=5" as gbytes=512 plus an
opt-in (%llu stops at the first letter and silently splits the field),
let a duplicated key win silently ("logprobs=3 logprobs=5" -> 5), and
pushed overflowing values through sscanf %llu, whose behavior on
out-of-range input is undefined (C11 7.21.6.2) -- a wrapping libc
folds 2^64+5 back to an in-range 5.
All three now reject through the existing BAD_REQUEST path: the
numeric gbytes field and the first key must be whitespace-separated,
each key may appear at most once, and the value is accumulated by hand
into a bounded int with an explicit range check, so no sscanf ever
touches it. One test case per defect added to the parse tests.
check_data_logprob_gaps.py audits that every generated DATA frame of an opted-in request carries a numeric value; a "nan"/"inf" lp (degenerate logits, e.g. an all -inf row after grammar masking) IS a carried value, but the check treated it as a failure -- conflating "the channel skipped this token" with "the value it carried is non-finite". Non-finite values now count as PRESENT and are FLAGGED in the report without failing the audit; whether the engine should serialize such rows differently is a server-side (U7b) question, deliberately not decided here. A true gap (a legacy 3-field DATA frame) still fails exactly as before.
…ap coverage weight_at (backend_cuda.cu) used to end in an unguarded int2 fall-through: any format without a device decoder was silently read as 2-bit values and returned plausible-looking numbers, while the CPU twins (qt_addrow, qt_matvec_rows) refuse the same input loudly. Pin the supported-format truth table in the host header as coli_cuda_weight_at_supported() so the host gates and the device backstop agree by construction, and cover it from both sides: - tests/test_cuda_fmt_guard.c: host-only truth-table test, runs in the plain C suite with no CUDA toolchain (same arrangement as metal_fused_fmt_ok). - tests/test_cuda_fmt_trap_cuda.cu: exercises the device-side __trap() refusal on real silicon under make cuda-test; re-execs itself per probe because the trap poisons the CUDA context.
repack_fp8_passthrough.py previously minted only routed experts and o_proj; its output directory was not standalone-loadable. Extend it to emit the full family: - kv_b_proj, norms, and io tensors alongside experts and o_proj (the minted kv_b_proj shard loads today but must not be used for batched-path decode until fmt=8 absorb decode lands; the tool's docstring states this explicitly); - aux files (config.json, tokenizer.json) copied into --outdir, closing the standalone-load gap; - canonicalized __metadata__ key order, so identical inputs produce byte-identical shards (pre-existing safetensors nondeterminism, present since the tool's original fmt7 form). Test coverage: test_fp8_repack.py grows the per-tensor and metadata determinism cases, test_fp8_repack_full_family.py (new) covers the full-family manifest and aux-file emission, and test_fp8_e2e_repack_load.py runs the minted directory through the real C loader end to end.
c/tools/README.md: describe the repack tool's full-family emission (experts, o_proj, kv_b_proj, norms, io) and the aux files it copies into --outdir. docs/FORMATS.md: document the fp8-e4m3 container-level declaration, the stamped collision shape, and the decode constraints on the minted kv_b_proj shard.
…falling through to int2 weight_at branched on fmt 0/1/2/4 and fell through to the int2 decode for everything else, so any format without a device decoder -- fmt=5 (int3-g64), fmt=6 (E8/IQ3), fmt=8 (fp8-e4m3), or whatever lands next -- was silently read two bits at a time and returned plausible numbers, while the CPU twins (qt_addrow, qt_matvec_rows) exit(1) loudly on the same input. fmt=3 is now an explicit branch and the fall-through is __trap(): abort the kernel and poison the context so the host's next cuda_ok() reports failure instead of the caller consuming fabricated values. absorb_fmt_ok now reads the shared coli_cuda_weight_at_supported predicate (already in backend_cuda.h on this branch) instead of restating 'fmt <= 4', which also admitted negative fmt values that would then have fallen through. This is the product half of tests/test_cuda_fmt_trap_cuda.cu (device-side trap coverage, already on this branch) and tests/test_cuda_fmt_guard.c (host truth table). No fmt=8 decode support is added here: weight_at still refuses fmt=8, exactly as the trap test asserts.
…nputs Three audit findings on repack_fp8_passthrough.py, one fix round: - RESUME: the progress manifest trusted os.path.exists, so a truncated or mutated "completed" output shard resumed as done, and an input shard mutated between runs went undetected. Manifest entries now record the emitted shard's size+sha256 (re-validated on resume; a mismatch re-emits to the recorded name, byte-identical for unchanged inputs, handing its stamp share back to the budget first) and the input shard's size+mtime_ns fingerprint (a mismatch aborts with InputChangedError: resuming would mix two sources into one container). A pre-fix string-schema manifest refuses loudly (ResumeManifestError) instead of being trusted unverifiable. Output shards get whole-file sha256 (the evidence class repack_rans.py's manifest already records); inputs get size+mtime_ns because every resume re-checks every input and hashing hundreds of GB of source per resume is the cost sha256 was rejected for on that side. - MALFORMED REPACK-KIND TENSORS: a repack-kind tensor with the wrong dtype or a missing _scale_inv sidecar fell through both selection predicates and was silently DROPPED -- an exit-0 mint with a missing weight. Real mode and --dry-run both refuse loudly now (MalformedRepackTensorError), naming the tensor and the exact defect. - CANONICALIZER SLOT-FIT: the header slot-fit guard was a bare assert, compiled out under python -O -- where an oversized rewrite (non-ASCII bytes json.dumps re-escapes longer than the raw UTF-8 the Rust writer stored) silently overwrote tensor payload bytes. Now a real ValueError, with a test that runs the corruption case under -O and asserts the file stays untouched. test_fp8_repack.py: 11 new cases (fp8 modules 47 -> 58), each verified to bite on the pre-fix tool.
…tation The full-family docs commit (855c292) claimed these pieces for c/tools/README.md and docs/FORMATS.md but did not include them. Now they exist: the standalone-loadable output family (fp8-repacked residents and routed experts, raw BF16/F32 pass-through for norms/router/embed/lm_head, and the config.json/tokenizer.json/generation_config.json aux copies, with the index.json non-copy rationale), and the constraint that a minted kv_b_proj loads today but must not serve batched-path decode until fmt=8 absorb support (f8/absorb-fmt8) lands -- a loud crash, not a silent misread, until then.
The trap test hardcoded per-fmt expectations ("fmt=8 must trap"), which
pins a moment instead of the invariant: the fmt=8 absorb-decode work
(f8/absorb-fmt8) adds a weight_at fmt=8 branch and flips
coli_cuda_weight_at_supported in the same commit, so on a tree where that
PR lands first, the frozen expectation fails on hardware with nothing
actually wrong.
Each probed fmt's expectation is now derived at run time --
coli_cuda_weight_at_supported(fmt) ? decoded : trapped -- so the test
asserts that weight_at's device dispatch AGREES with the predicate the
launch-site gates consult, in both directions: a predicate-false fmt that
decodes means the trap fall-through is gone; a predicate-true fmt that
traps means the promised decode branch is missing (or the harness cannot
tell the verdicts apart -- the failure message now names both readings).
The probe list is unchanged: both sides of today's truth table plus the
negative/out-of-range values the old `fmt <= 4` gate admitted, which
stay probed no matter what the predicate grows to support. fmts 0 and 3
double as the child-harness control, as before.
…build backend_gpu_compat.h maps only the CUDA runtime names backend_cuda.cu itself uses. The fmt trap test additionally uses cudaMemset (the backend memsets asynchronously only) and cudaErrorUnknown (the backend never names the generic error), so hipcc -- which compiles this file in the HIP syntax-check lane via gpu-compile -- finds them undefined. Alias the two locally per vendor in the test, the same arrangement tests/test_weights_owned_cuda.cu uses for its own off-surface names, rather than widening the product header for a test-only need.
weight_at's undecodable-format backstop calls __trap(), a CUDA intrinsic with no HIP name-twin, so the HIP syntax-check lane fails compiling backend_cuda.cu (rocm clang: use of undeclared identifier '__trap'). Map it in backend_gpu_compat.h's HIP arm -- the header's charter is that every platform difference lives here and backend_cuda.cu stays untouched, and it already maps a device intrinsic the same way (__syncwarp -> __syncthreads). HIP device code aborts via abort(), the mapping hipify itself applies. NVCC never sees the define, so the CUDA-side semantics the fmt trap test pins (kernel aborts, context poisoned) are unchanged; under HIP this makes the refusal compile, and its behavior on AMD silicon remains unverified by this repo's tests, as the header now says.
Two fixes to the microbenchmark, both of which change what it measures. 1. Block size is parsed with atol(), so any fraction of a megabyte truncated to 0. GLM-5.2's experts are ~19 MB so this never bit, but a fine-grained MoE expert (3 * d_model * moe_intermediate_size * bits/8) lands between 0.2 and 1 MB, and that whole range was inexpressible. Now atof(), rounded up to 4096 because O_DIRECT wants a sector-aligned length. 2. Every thread shared one fd. On Windows compat_open_direct() returns a SYNCHRONOUS handle (FILE_FLAG_NO_BUFFERING without FILE_FLAG_OVERLAPPED), and ReadFile on a synchronous handle serialises behind the file object lock, so [threads] had no effect at all and every Windows measurement was really queue-depth 1. compat_pread() is fine; the sharing was the problem. Each thread now opens its own fd. Measured on a Phison 512 GB NVMe, Windows 11, 768 KiB blocks, O_DIRECT: before 1T 1.09 2T 1.07 4T 1.07 8T 1.02 16T 1.02 GB/s after 1T 1.02 2T 2.31 4T 2.91 8T 2.78 16T 2.58 GB/s Cross-checked against an independent Python harness using CreateFileW with FILE_FLAG_NO_BUFFERING and per-thread handles, which gives 2.85-2.95 GB/s at 4-8 threads on the same file: the two agree after the fix and disagree by ~3x before it. Backward compatible: an integer argument behaves as before (19 -> "19 MB"), the default is unchanged, and tools/datapoint.py's IOBENCH_RE only captures the "-> N GB/s" field, which is untouched. Linux and macOS are unaffected in behaviour -- positional pread already concurs there -- but a private fd is correct on those platforms too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An expert's on-disk size is fixed at conversion time by the model's geometry, and the engine reads one expert per request, so the architecture decides where on the drive's block-size curve the streaming path operates. That curve is not flat on consumer NVMe. Measured on a Phison 512 GB NVMe (Windows 11, O_DIRECT, 128 reads, median of 3), against a plateau of 3.02 GB/s at >= 1.5 MB: 192 KB -36% queue-depth-1 penalty 4.33x 384 KB -17% 2.98x 768 KB -10% 2.15x 1.5 MB -3% 1.86x 12 MB +1% 1.20x Both effects are graded and monotone in expert size: smaller experts lose bandwidth outright, and lose proportionally more of it at low queue depth. That is relevant to #441 (PILOT prefetch at queue depth 1), whose cost is ~4x for a fine-grained container and ~1.2x for GLM-5.2's ~19 MB experts. One drive, one platform -- the knee position depends on the controller, NAND and whether the drive has DRAM, so this needs replication before it means anything general. The script exists to make that cheap: it needs no model download and no engine build beyond iobench itself, and prints a ready-to-paste markdown table. Stdlib only, like tools/datapoint.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
machine_info() hardcoded ram_gb=8.0 on anything that is not darwin or linux, and evict_cache() sizes its eviction write from it: on a 128 GB Windows box the tool wrote 9 GB, evicted nothing, and published the warm decode as the cold figure — with nothing in the output to say so (#1042). Add the win32 branch from the report: GlobalMemoryStatusEx for the physical RAM (checked, so a failed probe falls back to the old 8.0 instead of sizing the write from zeroed fields) and the registry ProcessorNameString so the datapoint carries the marketing CPU name rather than the family/model string. Stdlib only. The portable fallback now also announces the RAM+1GB temp-file write before starting it — correct behaviour, but surprising at 129 GB on a big box, and --no-evict already exists for whoever wants out. Probe fix suggested by @Unknown-Findout in #1042; tests mock the two win32 probes so they run on any host.
The test carried two unrelated assertions, and ci.yml already said the split was worth doing but not silently from a CI change: hit-rate equality across two greedy replays (a real determinism check) and a tok/s-within-25% bound over a ~15 ms tiny replay (a wall-clock check that measured 38.8% between identical runs on a quiet box, and fails 4/5 runs on clean dev under WSL2 — one scheduler stall dominates the whole run). Now they are two tests. test_cpu_vs_cpu_determinism keeps the name, the docstring's threat model (stray threading, uninitialized state) and only the hit-rate assertion — verified stable 5/5 on the same host that flaked before — and joins the CI list, which previously had to exclude it wholesale to avoid the timing half. test_cpu_vs_cpu_tok_s_stability holds the wall-clock bound, documents that it is noisy on shared or virtualized hosts, and stays out of CI.
…ayers The C engine (deepseek_v4.c) only requires compress_ratios to have at least num_hidden_layers entries and ignores trailing extras, but the python _dsv4_geometry check demanded an exact match. Models such as ds-v4-flash ship 46 ratios for 43 layers, making every doctor/plan run fail. Relax the check to >= and only iterate the first num_layers entries to mirror engine behavior.
The recipe exists in deepseek-v4.md (rows16 vs reference accumulation order, autopin history selecting between them) but #1136 shows a careful reporter reconstructing the whole mechanism experimentally because ENVIRONMENT.md — where the knobs are looked up — never points at it. Add the pointer next to the engine's usage-save note, updated for USAGE_SAVE now covering V4 too.
…i matching, and dynamic RAM cap
…ith the GPU tier The #1097 default (9 lanes, 1.41x CPU decode) measured on the GPU tier's reference box (RTX 5080, 2x NVMe, v1.7.0, 3324-token prompt, 48 greedy tokens, two interleaved runs per config): decode 33.9/32.4 s vs 35.3/33.6 s (~4 % — the VRAM mirrors and RAM cache absorb most decode misses), but prefill 126.5/127.6 s vs 114.9/116.4 s (~10 % slower — the loader threads contend with the MoE expert bank's parallel O_DIRECT refill for the same NVMe queues). Record that next to the knob and in the canonical serve line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kvb_all materialises H*(qk_nope+v_head) floats for every context token in one buffer — 30.1 GB at ctx 262144 — and cap_for_ram reserves that permanently even though the buffer only exists during prefill. That reservation is what drives the expert cache to its cap=1 floor at long context (5.46 -> 0.86 tok/s in the issue's measurements): the process pays 30 GB of resident budget for a transient. Above KVB_FLASH_MB (default 2048), rebuild kv_b for KVB_TILE_MB worth of tokens at a time and fold scores/values through an online flash-style softmax. The rebuild total is unchanged (one matmul pass over the context), scores and values keep their t-order; only the softmax normalisation is incremental, so output can differ from the one-shot path by rounding — the same kernel-family divergence class as the CUDA/Metal attention arms (#510). Peak transient drops from Tk*kvb_dim*4 to one tile plus 2 floats per (row, head), and the cap_for_ram reserve mirrors the same trigger, freeing ~28 GB of budget back to the expert cache at ctx 262144. DSA rows keep the one-shot path: the top-keep list is scanned in its two-band order and is not tileable by t-range. KVB_FLASH_MB=0 restores the old behaviour everywhere; KVB_FLASH=0/1 is the A/B switch. Verified: glm_tiny TF oracle 32/32 token-exact under the tiled path, both single-tile and (with the row floor lowered locally) 8-tile with causal clipping across every boundary; make test-c green.
Under DSA the score/value loops read only each row's top-keep list, yet the one-shot matmul rebuilt all Tk rows — the same 30 GB paid for a top-k read. Past the KVB_FLASH_MB ceiling, rebuild only the union of selected rows (rows with ns==0 scan their full causal range and force those rows in): per-row rebuild is independent and the read order along each tlist is unchanged, so this path is bit-identical to one-shot, unlike the dense tiling. Falls back to one-shot when the union does not at least halve the buffer, or on allocation failure. Verified: dense paths unchanged (tiny TF oracle 32/32 both modes), zero-warning build, make test-c green. The gather branch itself needs a DSA-active checkpoint to execute, which this box cannot run — logic argued in the PR, verification requested from a 4xA6000 host.
docs(v4): V4_LOADER_LANES — keep the 9-lane default for the CPU path, set 3 with the GPU tier
docs: cross-reference the V4 reproducible-run recipe from ENVIRONMENT.md (#1136)
test: split the tok/s bound out of test_cpu_vs_cpu_determinism
fix(tools): probe real RAM on win32 in datapoint.py (#1042)
attn: tile the prefill k/v reconstruction past a ceiling (#768 cause 2)
perf(dsv4): overlap hybrid fill DMA with host compute; pin slabs on Linux
The #1151 rebase gated the tiled (flash) and DSA-gather reconstruction arms off under quantized KV, keeping #553's one-shot + staging design untouched — correct for the merge, but it meant long context and KV quantization could not compose: the 30 GB kvb transient is independent of KV width, so a KV8 run at ctx 262144 still paid the one-shot buffer. Both arms now dequantize the latent rows they need through the same per-row codecs the one-shot path uses: the flash arm stages one tile's worth of Lc rows (tile-sized f32 buffer, ~8 MB at the default tile) before its kv_b matmul, the gather arm dequantizes exactly the union rows it rebuilds, and the roped-key reads inline the one-shot path's three-way branch so each representation keeps its exact accumulation order. The f32 paths are byte-identical to before — the branches sit on g_kv8/g_tq only. Verified on glm_tiny TF oracle: f32 default 32/32 (unchanged); KV8 and KV_TQ under KVB_FLASH=1 score 30/32 and 23/32 — identical to their one-shot scores — in both single-tile and (row floor lowered locally) 8-tile runs; make test-c green, zero-warning build.
The cross-format alignment discussion in #594 asked for exactly this: per-record {logical, encoded payload, framing} bytes plus physical placement, so byte amplification can be verified on real data instead of inferred from the achieved-ratio average. repack_rans.py --census <dir> walks an already-minted output directory read-only and writes rans-census.json: per record the byte-exact logical length (ceil(n_symbols/2)), the summed stream payload before padding, the framing around it (round16 header + payload pad), the absolute shard offset and extent, and an explicit alignment_bytes=0 — records pack back to back in the safetensors data section; O_DIRECT alignment is the reader's page window, not a property of the stored bytes, and a cross-format table wants that as a stated zero rather than a missing column. Any record whose extent is not exactly header + padded payload is flagged UNATTRIBUTED_BYTES and fails the run. Test: full attribution per record against the synthetic fixture corpus (extent closes to header+payload+pad, offsets point at real record starts, per-shard sums close, alignment stated as zero).
platform.release() says '10' on Windows 11 — Microsoft never moved the internal version off 10.0, so the build number is the only tell: 22000 and up is Windows 11. Every Win11 datapoint in the tracker was being filed as Windows 10 (#1042 follow-up, reported by the same tester who confirmed the RAM probe on real hardware). Tests cover both sides of the 22000 cutoff.
One amax per 512-dim latent row lets a single outlier dilate the quantization grid for the whole row. FlashMLA's fp8 KV cache stores one f32 scale per 128 latent elements for exactly this reason, and the #1140 attribution experiment pinned KV8's tiny-oracle flips to the latent value path — the component grouped scales tighten. KV8_GS=<n> puts one scale per n latent elements (default 0 = per-row, byte-identical to current behavior; the _gs entry points collapse to the per-row functions and the test pins that equivalence). RoPE rows keep per-row scale — qk_rope is 64 and was not implicated. Measured on the tiny oracle (kv_lora=32, so the group count maps to the real 512-dim ladder): f32 32/32 KV8 30/32 (per-row, as merged) KV8_GS=16 30/32 (2 groups) KV8_GS=8 29/32 (4 groups — near-tie flips are coin-flips along the way, not monotone) KV8_GS=4 32/32 (8 groups — full recovery) Byte cost at the real geometry: GS=128 (FlashMLA's choice, 4 groups) adds 12 B per token-layer over per-row, ~0.6% of the fp8 row. No .coli_kv format for grouped scales yet: KV8_GS forces KVSAVE=0 with a notice, same interim as KV8_ROPE would have; the v2->v3 bump is the follow-up once a granularity default is picked on a real checkpoint. test_kv_fp8: grouped round-trip with an outlier confined to one group (other groups' grids must stay tight, and grouped must beat per-row off the outlier), nscale arithmetic, and gs=0 bit-equivalence.
The fill-once Vulkan routed-expert tier uploads from freshly-read RAM
slots INLINE on the decode thread, capped at a fixed K3_VK_UP=8 per
step. A fixed count is wrong in both directions: on a slow bus eight
uploads stall the step, on a fast one the tier fills needlessly slowly
and decode keeps paying CPU matmuls for experts that could already be
resident.
K3_VK_UP=auto (opt-in; a number or the default 8 behave exactly as
before) replaces the count with a measured budget:
budget = fraction * step_seconds / upload_seconds
with both rates live EMAs (decode steps only), fraction from
K3_VK_FILL_FRAC (default 0.25), clamp [1, 64], and the legacy cap while
either rate is unmeasured -- enabling auto can never start worse than
the default. The policy is pure arithmetic in the new hybrid_split.h,
shared with the DSV4 hybrid split (deepseek_v4_hybrid.h stays as a
shim), and unit-tested GPU-less: legacy fallbacks, the budget formula,
both clamps, monotonicity in step time.
Visibility: the startup banner reports auto, and both the CLI decode
summary and the serve per-turn line print the current cap with the two
EMAs, so a single log shows what the policy decided and from what.
Validated end to end on Lavapipe (software Vulkan): default and auto
are token-identical over 24 tokens on the tiny fixture, VK matches the
CPU engine token for token, and the adaptive cap behaves as designed --
with memcpy-fast uploads (0.01 ms) under a 12 ms step it opens to the
ceiling and the tier fills within the first tokens instead of 8/step.
make check green; K3 tiny vendor oracle + checkpoint suite green.
fix(tools): label Windows 11 correctly in datapoint machine_info (#1042)
tools: per-record layout census for rANS containers (#594)
KV8: grouped latent scales (KV8_GS) — FlashMLA's fp8 cache geometry
perf(tune): persist rotating workloads and tune V4 loaders
attn: KV8/KV_TQ awareness for the flash and gather kvb arms (stacked on #1151)
perf(k3): measured adaptive expert-tier fill (K3_VK_UP=auto)
# Conflicts: # c/Makefile
perf(tune): measure safe RAM and cache caps
feat(tools): full-family FP8 (e4m3) container mint with content-validated resume
Engine open used to construct the safetensors index SIX times: once retained for the engine, once retained for the expert store, and four throwaway builds inside the resource planner, the dense inventory and the head-cache probe/load. Each build opens every shard, preads and parses every header and hashes every tensor name -- on a 141-shard checkpoint that is ~100k names five more times than needed, plus three parses of config.json and three full per-layer plan+validate walks. - The planner, inventory and head-cache paths now take the engine's retained target_index (and its parsed config); the two retained builds remain, the four throwaways are gone. One layer walk yields both the per-layer maximum and the dense total, replacing the separate inventory pass entirely. - A new stderr line, `v4_open index=%.2fs plan=%.2fs`, makes the open phases measurable at last: datapoints report "engine load" from time_to_first_token, which folds in session create and the whole first prefill, so open-phase wins were invisible before. - v4_fp8_pack_rows8_inplace -- a full second pass over every dense FP8 byte at load -- swaps its scalar transpose (eight cache-line reads per 8 bytes written) for an 8x16 unpack-chain transpose with 16-byte loads and stores. Byte-identical order by construction, and the token-exact tiny oracle gates it end to end. - The expert manifest's records array drops a redundant calloc (build_record zeroes each record before filling it). Not included, on purpose: the pin-after-populate and mint-layout items sketched for this slot were checked against the code first -- colibri's startup pin path already populates before wiring (explicit comments at colibri.c:2738 and 9271), and the safetensors writer does not guarantee payload ordering, so a layer-contiguous mint flag would promise what it cannot deliver. What the code disproved was dropped. make check green; V4 tiny oracle token-exact; every amalgam unit compiles under -Werror=implicit-function-declaration with and without COLI_V4_GPU_TIER.
perf(dsv4): build the shard index once per engine open
…I_CUDA_MOE_DOUBLE=1) Opt-in second full-layer expert bank: while the compute stream chews layer L from the active bank, a worker thread uploads layer L+1's complete expert set into the spare bank over the device's aux stream, starting before L+1's routing is known. On the layer switch the banks swap and the per-expert valid map travels with the swap, so partial prefetches are topped up by the existing route-aware refill instead of thrown away. - deepseek_v4_bank_pair.h: pure swap/legacy + prefetch-target decisions, unit-tested GPU-less (tests/test_v4_bank_pair: bootstrap, segment restart, worker failure, last layer, 61-layer sweep) - backend_cuda_dsv4.cu: bank upload refactored onto a caller-chosen stream (compute or aux), DeepGEMM scale packing and pointer-table writes ride the same stream, single drain before return; dsv4_cuda_expert_bank_upload_aux added - backend_loader_dsv4.c + dsv4.def: aux upload exported, resolved as OPTIONAL so older DLLs keep the tier; the engine detects the missing export on the first failed upload and permanently falls back to single-bank - deepseek_v4.c: worker holds at most one store lease at a time (bounded pin slots), L+1 lookups live in L+1's own slot partition so the computing layer is never evicted; every failure path (second-bank allocation, aux upload unavailable, fully failed layer) degrades loudly to today's single-bank behaviour; swap/prefetch counters printed at release Default (env unset) is byte-identical to the current single-bank path.
perf(dsv4): double-buffered expert bank prefetch for GPU prefill (opt-in)
perf(cache): index GLM, OLMoE, and Qwen expert slots
The README had drifted behind the engine in ways that were factually wrong, not just stale: - The DeepSeek V4 requirements row said the CUDA tier needs sm_80+; since the portable-pre-ampere work it runs on Pascal and Turing (GTX 10 / RTX 20), built with CUDA_ARCH=portable-pre-ampere NO_TC=1. Fixed in the table and in the V4 section. - The roadmap still listed Kimi K2 and Qwen3 MoE as future work while Kimi K3 and Qwen3.6 are documented as running two sections earlier. Rewritten around the six families that run today. - Version strings in the sample banner and the release-archive example were several releases old. New since the last README pass, now documented: - Kimi K3 recurrent-state checkpoints (COLI_K3_CKPT, COLI_K3_CKPT_DIR) and the measured Vulkan tier fill (K3_VK_UP=auto), plus the token-exact KDA/MLA oracle that gates them in CI. - The two opt-in DeepSeek V4 GPU levers (DSV4_HYBRID, COLI_CUDA_MOE_DOUBLE), labelled as experimental and looking for community numbers; defaults stay byte-identical. Translations (it, zh-CN, zh-TW) updated for the factual points they carry: version strings, the roadmap paragraph, and zh-CN's DeepSeek V4 section that still described the engine as an experimental CPU-only path.
docs(readme): catch the README up with what already runs
…ows DLL Since the V4 CUDA tier landed, dsv4_cuda_available() answered False on every non-Windows platform: it only looked for coli_cuda_dsv4.dll next to the engine. On Linux the tier is linked straight into the binary by nvcc (-lcudart), so a perfectly good `make deepseek-v4 CUDA=1` build was refused by --gpu/--vram with a hint about building a DLL that does not exist on Linux (#1219). v1.6.1 predated the gate, which is why the report says the regression starts at 1.7.0. Detection now mirrors what cuda_binary() already does for GLM three lines below: on Linux, ldd on the V4 engine binary must list libcudart (or libamdhip64) and not as "not found"; on Windows the DLL probe is unchanged. The error hint names the right build per platform: make deepseek-v4 CUDA=1 on Linux, the DLL target on Windows. Unit tests cover both platforms: CUDA and HIP links accepted, CPU build and broken links rejected, missing engine rejected, DLL present/absent, and the per-platform hint text.
fix(coli): detect the Linux DeepSeek V4 CUDA build, not just the Windows DLL
Require the CUDA runtime used by the in-tree backend, avoid masking unexpected detector bugs, and report unsupported host platforms accurately.
fix(coli): harden DeepSeek V4 CUDA detection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
60 pull requests since v1.7.0. The cycle's spine: every engine now finds a
resident expert in one probe instead of a scan, prefill stopped re-reading
bytes it already had, and the DeepSeek V4 CUDA floor dropped from Ampere to
cards from 2016. Everything risky is opt-in; every default path is covered by
token-exact oracles.
Expert lookups: O(1) on every family
cost made independent of the configured cache size.
Kimi K3, then to GLM, OLMoE and Qwen3.6: a resident hit is 1 probe where
the legacy worst case was 44 to 219, validated by model-free tests with a
deterministic probe counter. Every lookup revalidates the slot identity, so
stale bookkeeping can only become a miss, never wrong weights.
shared-expert work; the LRU used to re-read the same experts several times
per layer under prefill's expert-major sweeps.
loads coalesced; FP4 expert matmuls batched for prefill and specialized per
phase; fp8 attention weights decoded with SIMD instead of per-element
gathers.
DeepSeek V4: a wider GPU tier
CUDA_ARCH=portable-pre-ampere NO_TC=1bringsthe whole CUDA tier to Pascal and Turing (GTX 10 / RTX 20 series); the
loader floor drops accordingly.
DSV4_HYBRID=1: on a VRAM-tier miss the missingexperts are split between the GPU fill branch and the CPU branch, sized by
bandwidths measured at runtime, with the fill DMA overlapped against host
compute and host slabs pinned on Linux.
COLI_CUDA_MOE_DOUBLE=1: a second full-layer expertbank is prefetched on a dedicated stream while the current layer computes,
and the banks swap at the layer switch. Falls back to single-bank when VRAM
is short.
times, the config parsed once instead of three, and the fp8 row transpose
vectorized bit-identically; a
v4_openstderr line reports the split.V4_LOADER_LANESwith andwithout the GPU tier.
Both
DSV4_HYBRIDandCOLI_CUDA_MOE_DOUBLEare experimental: defaults arebyte-identical with the envs unset, and they are looking for community
numbers on real cards.
Kimi K3: checkpoints, a vendor oracle, tool calling
paths token-exact against Moonshot's reference implementation, in CI,
without torch or network.
COLI_K3_CKPT=N,disk-parked via
COLI_K3_CKPT_DIR): an edited or follow-up prompt restoresthe deepest surviving checkpoint and re-prefills only the tail instead of
replaying the whole conversation through the SSM layers.
K3_VK_UP=autosizes the Vulkan expert-tier upload frommeasured bandwidth instead of a fixed guess.
Moonshot's reference renderer.
KV cache quantization (@ZacharyZcR)
and
KV8_GSadds grouped latent scales following FlashMLA's fp8 cachegeometry.
(cause 2 of KV cache is stored in fp32 and kvb_all materialises the decompressed K/V (x3.2 vs llama.cpp) #768).
coli tune: measured, for every engine
just GLM.
of deriving them from totals.
--capisabsent; rotating workloads persisted and V4 loaders tuned; Metal reported
correctly in the plan label on Apple Silicon.
RAM_GBis clamped to the memoryactually available ([Bug]: RAM_GB is not clamped to the #653-corrected MemAvailable on unified memory; CAP_RAISE then inflates against it and the run is OOM-killed mid-prefill #759).
Fixes across the fleet
--vram check only knew the Windows DLL and refused every valid Linux
make deepseek-v4 CUDA=1build ([Bug]: CUDA build not recognized in Linux since 1.7.0 #1219, a 1.7.0 regression); detection nowreads the binary's actual CUDA linkage, rejects lookalikes, and names the
right build per platform. Reported by @ANBAL534.
Linux.
qgsat the callsites; the webUI renders the reasoning stream instead of dropping it ([Bug]: deepseek_v4 "coli serve" frontend doesn't stream (or ever show) thinking tokens #1148).
for the default tier and skips the discarded final decode.
make installbuilds qwen36 first.compress_ratioslonger thannum_hidden_layersaccepted.
Tools, bench, CI, docs
content-validated resume; a per-token logprob channel, echo prefill
read-out, and token-ID prompt intake.
layout census for rANS containers; datapoint fixes for Windows 11 labeling
and win32 RAM probing; the tok/s bound split out of the determinism test;
V4 docs cross-references.
eviction write.
teardown; persistent rotating-prompt bench datapoints; iobench sub-MB
blocks and a Windows
[threads]fix.METAL=1.the six families, K3 checkpoints, and the experimental V4 GPU levers.
The version bump to 1.8.0 is included (
c/version.py).