Conversation
A discovered device and a placement budget are not the same fact, and the
planner had no way to say so: `free_bytes` was mandatory and numeric, and
every consumer treated "a GPU is present" as "a GPU may be planned against".
That is fine while free VRAM *is* the budget, which is true of a discrete
card. It stops being true on a part where the GPU and the host share one
physical pool and the runtime's free figure describes pages the RAM tier is
already counting.
So `free_bytes` may now be None, meaning "not qualified as a budget on this
platform" — deliberately NOT the same claim as 0, which is a measurement
("the card is full") and keeps every behaviour it has. plans_placement()
tests `is not None` rather than truthiness so the two cannot collapse.
Only qualified devices reach the decisions: the VRAM budget, the shared-pool
`unified` flag, `_auto_tune`, the bottleneck classification, the
"VRAM already in use" warning (which divides free by total, and would
otherwise tell the user to stop an engine that is not running), and the
COLI_GPU/COLI_GPUS device list — that one is an instruction, not an
inventory. Everything discovered is still reported, marked `(identity only)`
in `coli plan` with a warning naming it, so the hardware does not silently
vanish from the output.
Also makes the no-device line backend-neutral. "no NVIDIA device detected" on
an AMD host was the same vendor assumption #903 removed from doctor; this is
the line it missed.
No numeric-`free_bytes` behaviour changes: a plan built from a discrete card
produces the same tier, the same tuning and the same environment as before.
`rocm-smi` is a Linux tool. Neither the Windows HIP SDK installer nor a source build ships it, so `_discover_amd_gpus` found nothing on Windows and every AMD host there was planned as if it had no GPU: no device in `coli plan`, "no supported GPU detected", and `--gpu N` failing on hardware that works. `hipInfo.exe` is what both shipped SDKs do provide, and it sits in the same directory as `amdhip64_7.dll`. It comes from the HIP environment already required to build the backend, so this adds no dependency. Lookup stops at the first hit: COLI_HIP_RUNTIME_DIR, then %HIP_PATH%\bin, then PATH. COLI_HIP_RUNTIME_DIR leads because it names the runtime the loader will actually bind, and hipInfo lives beside it — on a host with two HIP installs, which is common, a stale machine-wide HIP_PATH must not describe the hardware through a runtime the engine will not load. Nothing is hardcoded. A block missing a name or a memory total produces no device rather than one completed with zeros; zeros would read as measurements. Missing, failing or unparsable hipInfo yields no device at all. Devices are recorded identity-only: `isIntegrated: 1` maps onto the `unified_memory` flag #903 introduced, `gcnArchName` is kept (Windows needs an explicit HIP_ARCH, so the arch is worth showing), and `free_bytes` is None. That last part is the deliberate one. hipInfo DOES report free memory. On the validated gfx1151 host it reported 89.24 GB "100% free" while Windows had 59.3 GiB physically available — the same pages counted twice, ~30 GB apart, on a part where the GPU and host share one pool. Spending that as a VRAM budget would authorise an expert tier the machine cannot back, while the RAM tier is sized from the same memory. The value is parsed and discarded rather than stored, and docs/windows.md says so and shows the numbers. This changes only what Colibri turns on by itself. Every environment variable keeps working as documented. Qualifying a safe automatic budget on shared memory needs measurement on real hardware, and is left to a later change instead of guessed at here. The Linux rocm-smi path is untouched, and a test pins that.
#903 made the accelerator check backend-neutral in wording. The probe under it was still CUDA-specific: on Windows it required coli_cuda.dll beside the executable, whatever the host was actually built to load. A HIP_DLL host loads coli_hip.dll. So doctor reported "GPU runtime library is missing" on a working HIP install -- and that branch is a fail, which makes the whole report status `error`, not a warning. Verified against the engine built for the #788 validation: it carries coli_hip.dll beside it, and the old rule called it missing. Accepting either name would trade one wrong answer for another: a HIP host with a stray coli_cuda.dll left over from a CUDA build would pass while the backend it actually needs is absent. The binary already says which one it wants. backend_loader.c compiles exactly one basename in -- COLI_BACKEND_DLL is "coli_hip.dll" under COLI_HIP_DLL and "coli_cuda.dll" otherwise -- so read it out of the image and require that artifact. A binary carrying neither is not a DLL-split GPU build and keeps its existing CPU-only answer. This validates the host/artifact contract and nothing more. It does not claim the HIP runtime loaded, that runtime identity was verified, or that any GPU computed anything; those are separate contracts with their own checks.
The Windows bare-`coli chat` auto-enable reads discover_gpus() and writes
COLI_CUDA and COLI_GPUS into the engine environment itself, before
environment_for_plan() is ever consulted. The planner's own placement gate
therefore cannot cover this path, and a device discovered without a qualified
free-memory budget was enough to switch the GPU on.
With a Windows AMD device (free_bytes None) that produced a full auto-enable
and a sized expert tier from a budget nothing had measured:
[GPU] auto-enabled CUDA · AMD Radeon(TM) 8060S Graphics · 8.6 GB expert tier
COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=8.000
It does not fire today only because cuda_binary() still looks for
coli_cuda.dll on Windows and so reports false on a HIP host. That is a
separate defect and is deliberately left alone here -- but it means the leak
is one unrelated fix away from becoming live, and whoever makes that fix
would have no reason to know this gate exists.
Filter at the source with the same predicate the planner uses, rather than a
second local rule that could drift from it. Auto-enable is an automatic
placement decision; only a device whose memory is qualified as a budget may
drive one.
Explicit user intent is untouched: --gpu and --vram never reach this branch
(it requires `a.gpu is None and not a.vram`) and keep working exactly as
before, as does every Windows NVIDIA host, whose devices carry numeric
free_bytes and remain qualified.
The discovery commit justified free_bytes=None with a single reading: hipInfo
reporting 89.24 GB free against 59.3 GiB of Windows-available memory, called
the same pages counted twice. The measurement was real, but it came from one
session under a firmware configuration that was not recorded, and the
explanation went further than the evidence.
Four controlled observations since, each in its own rebooted session with
everything else held constant, varying the Armoury Crate shared-memory limit
across its full range:
~6 GB total 76.79 GiB free 76.63 GiB Windows visible 127.15 GiB
~32 GB total 76.79 GiB free 76.63 GiB Windows visible 127.15 GiB
~64 GB total 76.79 GiB free 76.63 GiB Windows visible 127.15 GiB
~123 GB total 93.00 GiB free 92.84 GiB Windows visible 127.15 GiB
Three settings spanning a twentyfold range gave the same reading; only the
maximum differed. At the minimum the reported total was ~12.8x the configured
limit. Windows-visible memory and the 0.50 GiB dedicated frame buffer did not
move at all, including in the run where the HIP figure did -- so the increase
was not taken from the host.
That is a better argument for free_bytes=None than the original one, and a
more careful one. It says what was measured and stops there: no claim that
hipInfo over-reports, that the figure is a fixed hardware property, that the
setting maps onto it, or that HIP free and host-available memory are the same
pages. Whether they are is exactly what a bounded allocation experiment would
have to establish, and nothing here has allocated anything.
Wording only. The AST of resource_plan.py is unchanged with docstrings
excluded; the parser, the planning boundary and free_bytes=None all behave
exactly as before, and the owner suites stay green.
The new Windows backend-artifact tests passed on Windows and failed on Linux
and macOS, six of them, all with the same shape:
{'linked': False, 'missing': False} != {'linked': True, 'missing': False}
'warn' != 'pass'
The helper faked both sys.platform and os.name to reach the Windows branch.
Faking os.name is the problem: pathlib picks its flavour from it at
construction time, so the Path() that cuda_linkage() builds from the fixture
path became a WindowsPath. On a POSIX host /tmp/xxx/colibri.exe turns into
\tmp\xxx\colibri.exe, which does not resolve, so the is_file() guard returned
early and every assertion compared against a false negative -- the
backend-marker logic was never reached at all.
Measured on Ubuntu 24.04 / CPython 3.12:
PosixPath /tmp/xxx/colibri.exe is_file() True
WindowsPath \tmp\xxx\colibri.exe is_file() False (os.name = 'nt')
So the helper now fakes sys.platform only, which leaves pathlib alone. That
needs the dispatch to key on sys.platform as well, because os.name is still
"posix" under the test and would otherwise send it down the ldd branch.
On every real host the two agree -- Windows is nt/win32, Linux and macOS are
posix/linux and posix/darwin -- so all three select exactly the branch they
selected before. Only the synthetic posix/win32 pair behaves differently, and
that pair cannot occur outside a test.
The alternative was to skip these tests off Windows. They are worth more than
that: the Windows backend contract stays reviewable by anyone, on any runner,
without the hardware.
No change to the backend-marker scan, the expected-DLL extraction, the
artifact check, the linked/missing semantics, or run_doctor's accelerator
status.
Replace the XOR-based hash (layer*C1 ^ eid*C2) with a linear-index multiply-and-shift (layer*256+eid) * golden-ratio >> 16. The old hash was uniform across the full 43x256 grid, but the hot subset (~12 experts per layer) clustered on one replica because the XOR didn't spread small inputs evenly. The fix maps each (layer,eid) to a unique flat index 0..11007 and multiplies by the golden ratio, so any subset splits evenly.
v4_read_direct_window only counted mirror (rep!=0) reads, hiding primary reads and making the split appear as ~33/67 when it was actually ~49/51.
The dual-SSD mirror added a ep (replica) parameter to v4_read_expert_record. The test assertion needs to reflect the updated call signature.
This is #772 rescoped. It was an engine plus its kernels and it would not start below 145.3 GiB of VRAM, which makes it a threshold -- the one thing this project does not ship. @jazir555 and @rafpigna said so, @JustVugg agreed, and the request was not negotiable: a GPU backend here holds the hot part and falls back for the rest. The split @JustVugg proposed is the right one. #165 landed the DeepSeek V4 engine with the streaming machinery and no CUDA at all; this was the CUDA with no tier. Neither runs the model on hardware anyone owns. So the engine file belongs to #165 and this keeps only the kernels: backend_cuda_dsv4*.{cu,h} dense matmul, batched attention, routed MoE dsv4_mhc.h, dsv4_quant.h the CPU-side formats the kernels consume tests/ three GPU tests, two pure-CPU unit tests tools/ oracles and probes the kernels are checked against Dropped from the old branch: c/deepseek_v4.c (#165 owns it), and the c/coli, c/openai_server.py, c/tok.h and docs/api.md edits, which belong to whichever engine lands rather than to a kernel PR. The kernels are self-contained by construction -- backend_cuda_dsv4.h includes only <stdint.h>, and the .cu only its own header and the CUDA runtime -- so they build and their CPU tests run with no engine present. That is what makes them usable as a tier: #165 calls in, nothing calls out. Makefile: the kernel rules only. deepseek_v4$(EXE) is gone; the .o rules, dsv4-cuda-test and the two header unit tests remain. DSV4_CUDA_OBJ is empty unless CUDA=1. VLLM_MHC/DEEPGEMM/FLASHINFER are marked UNSUPPORTED in place -- eight configurations, none compiled by CI, each needing an external checkout -- and DEEPGEMM's flag line appends instead of replacing, so it no longer discards $(CUDA_GENCODE) and -ccbin. Verified: colibri, deepseek-v4, cuda-test and bench-omp-grain all still resolve; dsv4-cuda-test resolves; test_dsv4_mhc and test_dsv4_quant build and pass on CPU. The GPU tests need a device and are compile-checked only.
…e gates' predicate
The load-time format notice covered kv_b_proj only, and had already drifted
from the gates it describes: both fused Metal decode gates carry
(kv_b.fmt==2 || (kv_b.fmt==4 && !g_moe_exact)), but the notice tested
fmt!=2 && fmt!=4 -- so under COLI_METAL_MOE_EXACT=1 a grouped-int4 (fmt=4)
kv_b closed both gates while the notice stayed silent: exactly the silent
CPU-fallback trap the notice exists to kill.
Type the per-layer format condition ONCE, in a shared predicate:
- metal_fused_layer_fmt_miss(l) returns a bitmask (METAL_FUSED_*) of the
8 fused-bound weight tensors whose format blocks the fused path: kv_b on
the gates' own two-format+mode term, q_a/q_b/kv_a/o and (sparse layers
only) sh_gate/sh_up/sh_down on the metal_fused_fmt_ok {1,2,3,4} allowlist.
- Both gates now consult it against the mask of tensors their kernel binds
(METAL_FUSED_ATTN_TENSORS / METAL_FUSED_LAYER_TENSORS). No behavior
change: the masked test is the same boolean the inline conditions computed.
- kvb_fmt_gate_notice becomes metal_fmt_gate_notice: one line per offending
tensor KIND (bounded at 8 lines, never per layer), sh_* counted over the
sparse-layer population, MTP head excluded, silent without Metal. Because
the notice consumes the gates' predicate, the two cannot drift apart again.
- g_moe_exact moves next to g_metal_enabled, outside #ifdef COLI_METAL (the
same hoist #587 did for g_metal_enabled): the predicate compiles on every
platform's test build; the env parse stays Metal-only, so the flag stays 0
elsewhere, which is correct there.
test_kvb_notice.c grows to a superset of the previous scenarios: per-kind
misses, multi-kind bounded output, all-pass/Metal-off/MTP silences, dense
sh_* exemption, predicate truth table and per-bit gate-mask membership in
both g_moe_exact states, mixed dense/sparse denominators, and the new
MOE-exact cases (fmt=4 kv_b notices under the mode, stays silent outside
it). test_fp8_load.c Part F re-pins the source-text wiring at the new shape:
7 metal_fused_fmt_ok(l->...) sites (all inside the helper) plus both gates'
predicate consultation. docs/FORMATS.md gains the format-consumer note for
the fused Metal decode path with anchors at this tree.
…rdening Review fixes on the widened fmt-gate notice: - The kv_b line's remedy was circular under COLI_METAL_MOE_EXACT: the converter's --group-size defaults to 64, so the suggested --kvb-bits 4 mints grouped int4 (fmt=4) -- exactly what the mode keeps off the fused path. The remedy clause is now selected on g_moe_exact: the plain suggestion outside the mode (unchanged), and under the mode the two real cures -- an ungrouped requant (--kvb-bits 4 --group-size 0 -> fmt=2) or, for a kv_b already at fmt=4, unsetting COLI_METAL_MOE_EXACT. Tests pin both variants (positive pins under the mode, a negative --group-size pin outside it). - The notice header claimed the fused gates never see the MTP head; false -- attention_rows evaluates the MTP row and its INT8 kv_b closes the gate there (carried, correct CPU fallback). Comment corrected to the actual rationale (main-layers-only by decision; a line about the INT8-by-design MTP head would be noise on every load). Comment-only: no gate change. - New accepted-behavior test case: runtime-quant dense configs (qt_alloc fmt=0 at bits>=16, fmt=3 at 3-bit) produce bounded, accurate output -- 8 lines all naming fmt=0, and exactly one kv_b line at fmt=3 -- so the behavior is claimed and pinned rather than accidental. - The test's stderr-capture seam now aborts loudly if dup/freopen fails instead of limping on with an undefined stderr. FORMATS.md anchors recomputed for the shifted lines.
4 tests against a mock SERVE engine (no checkpoint needed): - non-streamed tool call: DSML block parsed into OpenAI tool_calls - streamed marker suppression across 12 chunks - <tool_result> round trip rendered into the next prompt - plain chat unchanged when no tools are declared Complements the vendored primitives merged in #1003 with wire-level coverage. Validated live on a 284B V4-Flash running CPU-only in 32GB RAM (Contabo VPS), where the GPU-tested base hadn't been exercised.
…RECT, config-sized routing - Serve mode now prints the cumulative per-drive read split on stderr after every DONE (the CLI path printed it only at exit, so the mode that matters in production had no way to confirm the split — and with the GPU MoE bank refill streaming ~90 GB per large prefill, expert I/O is now on the hot path, not just cold start). - Direct-I/O availability is decided per ROUTED replica (coli_st_streaming_direct_available_rep): a mirror with an O_DIRECT twin no longer loses direct reads when the primary lacks one, and vice versa. Same COLI_V4_DIRECT gate. - The route hash uses the config's experts_per_layer instead of a hardcoded 256, threaded through coli_st_mirror_setup; the split stays exact for any future expert-count geometry. Engine builds clean (ARCH=native); test_deepseek_v4_dspark_source 9/9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # c/deepseek_v4.c
hot_pack_slot_locked() repacked pinned experts (three coli_fp4_pack_rows16_v10 passes + six memcpys over ~12.6 MB) while holding state->mutex on the miss path, serialising every other OMP fetch during batched prefill. Per issue #900 this is ~5.7 s of the v1.5.0 TTFT regression. Split the work: hot_pack_slot_prepare() runs the CPU-heavy packing into a private buffer lock-free (the slot already holds our reference, so its slab is stable), and hot_pack_slot_commit() — still under state->mutex — copies the packed buffers into the slab and flips policy->packed[]. The (slab layout, packed flag) pair therefore stays consistent for any reader, exactly as before, and the dominant cost leaves the lock. Verified: deepseek_v4.c builds clean (incl. COLI_V4_UNIT_EXPERT_STORE_HOT_ROWS16), V4 unit tests pass, and the token-exact tiny + prefix gates are byte-exact.
ci: run Metal backend tests on macOS
deepseek_v4: let the launchers delegate OpenMP team sizing to the runtime
# Conflicts: # c/deepseek_v4.c
feat(v4): the expert history lives in route_trace.h now — #700 completed
fix(CUDA): fmt=8/fmt=6 scale-byte accounting — rebase of #1100 by @monotophic
rss_guard lowers ecap after freeing slabs, but the freed slots stayed inside ecn[layer] and every victim scan picked them first: reusing a slab-less slot re-allocates its slab, so the cache grew straight back to its pre-guard size and the guard kept firing forever (#1034). Fold the three duplicated victim scans (demand-swap promotion, pilot_realload, pilot_uring_batch) into eslot_lru_victim and teach it the cap: an emptied slot (eid=-1, slab=NULL) is only reusable while the row's live-slab count is under ecap; at or over the cap the scan falls back to evicting a slab owner, which never grows memory. In-flight reservations (eid<-1) count as live since they are about to own a slab. Below the cap the empty slot is still preferred over evicting a resident expert, so steady-state behavior is unchanged.
USAGE_SAVE=0 is documented as a shared read-only switch, but only inkling and kimi_k3 checked it: the GLM engine's usage_save() wrote unconditionally (five call sites, two of them per-turn), olmoe never checked, and deepseek_v4 listened to a different, undocumented variable. A benchmark loop that trusted the flag silently contaminated the very profile PIN=auto ranks from (#1039). Move the check into rt_save() itself, where every engine's history write already funnels through — one definition, five engines, and any future engine gets it for free. The engine-local checks in inkling and kimi_k3 become redundant and are removed; a requested skip returns success so V4's cannot-save-history warning stays quiet. The V4-only COLI_V4_SAVE_USAGE alias keeps working and is now documented. test_route_trace.c gains the read-only case: rt_save under USAGE_SAVE=0 reports success and leaves the file untouched.
fix(serve/coli): make GPU-vs-fallback counters and chat status actually visible
Implement _olmoe_geometry in family_registry.py, following the engine's actual allocation in olmoe.c rather than the config's GQA hint: - K/V cache sized with num_attention_heads (not num_key_value_heads) because olmoe.c allocates m->K[i]/m->V[i] per q-head (olmoe.c:1019-1020, 1047-1048, 1097-1098, 1412-1413), fp32. - head_dim derived as hidden_size / num_attention_heads (olmoe.c:327). - workspace = 4 attention temporaries (q, k, vv, ctx) x S x hidden floats (olmoe.c:643, 662). Activate planner_id=olmoe_gqa in FAMILIES and drop the unsupported reason. Tests: 4 new cases covering engine parity, GQA-ratio sizing (16 q-heads vs 4 kv-heads still sizes from 16), linear scaling with context, and missing-key rejection. Full suite: 547 tests, 543 passed, 4 platform-flag false-failures on ARM64 (documented separately).
Implement _kimi_geometry in family_registry.py, following the engine's hybrid allocation in kimi_k3.c: - MLA cache (Lc/Rc) sized per non-KDA layer: context x (kv_lora + qk_rope) x 4 bytes (kimi_k3.c:1704-1706). Only the 24 gated MLA layers keep a positional cache that grows with context. - KDA recurrent state sized per KDA layer: kda_heads x kda_hd x kda_hd x 4 bytes (kimi_k3.c:691). The 69 KDA layers carry a fixed recurrent state that does NOT scale with context -- reported in fixed_state_bytes like GLM's indexer state. - KDA layer membership read from linear_attn_config.kda_layers (1-based indices, kimi_k3.c:552-555); everything else is MLA. - Workspace = max of the KDA forward temporary set (6*C*P + C*hd + C*H, kimi_k3.c:896-898) and the MLA set (qa + qv + ckv + gv + ctx, kimi_k3.c:992-994), since a single forward pass touches one layer type. Activate planner_id=kimi_hybrid in FAMILIES and drop the unsupported reason. Update the refuse-instead-of-inventing-zero test: kimi_k3 is now supported (only inkling and deepseek_v4 remain unproven). Tests: 6 new cases covering engine-parity hybrid allocation, KDA state context-independence, linear MLA scaling, workspace max selection, and fail-closed behavior on missing linear_attn_config / kda_layers / keys. Full family_registry suite: 22 passed; planner pipeline: 53 passed.
Implement _inkling_geometry in family_registry.py, following the engine's hybrid allocation in inkling.c: - KV cache per layer sized from the layer's kv heads and head dim (L_KV/L_HD, inkling.c:80-82; kv_alloc, inkling.c:1687-1699). Sliding-window layers are a ring of window rows (kv_ring_rows, inkling.c:1124-1125) -- at context > window that caps their growth, while the 1-in-6 global layers keep the full context. - Conv states (inkling.c:888-890, 1646): four depthwise short-conv states per layer (cs[0]/cs[1] kv-wide, cs[2]/cs[3] hidden-wide), each holding conv_k-1 history rows. Context-independent, so they go in fixed_state_bytes. - Workspace = peak of the attention temporary set (q, k, vv, rr, ctx, inkling.c:1141-1147) across the global and sliding layer shapes. - Audio tower (inkling.c:808-832): optional DMel encoder embedding table [mel_bins*mel_vocab, hidden] + RMSNorm added to fixed_state when audio_config is present. - Sliding membership follows the engine precedence: layer_types[] (fail-closed on length mismatch) > local_layer_ids[] > (i+1)%6 rule. Activate planner_id=inkling_hybrid in FAMILIES, drop the unsupported reason. Only deepseek_v4 remains unproven in the refuse test. Tests: 6 new cases -- engine-parity hybrid allocation, sliding ring context-capping, local_layer_ids override, audio tower fixed reserve, fail-closed on missing keys and on bad layer_types length. family_registry suite: 28 passed; planner pipeline: 53 passed.
…all 4 families Implement _dsv4_geometry in family_registry.py, mirroring the engine's own context_bytes() (deepseek_v4.c:1271-1284) exactly: - Fixed sliding-window ring: num_hidden_layers * sliding_window * head_dim * 4 -- a circular buffer that does not scale with context, so it goes in fixed_state_bytes. - Per-layer compressor state: ceil(context / ratio) * head_dim * 4 for every layer with compress_ratios[layer] > 0. - Per-layer indexer state: ceil(context / ratio) * index_head_dim * 4 for layers with ratio == 4. - Workspace mirrors the batched-attention scratch set (deepseek_v4.c:2741-2750): qa (q_rank), q (q_width), kv (head_dim), attended (q_width), oa (oa_width), norm (max(q_rank, head_dim)), with q_width = heads * head_dim and oa_width = o_groups * o_lora_rank. Activate planner_id=deepseek_v4 in FAMILIES, drop the unsupported reason, and empty the refuse-instead-of-inventing-zero test: all four families from issue #1066 now have proven Python geometry adapters. Tests: 6 new cases -- exact parity with engine context_bytes, ring context-independence, workspace scratch parity, fail-closed on bad compress_ratios length and missing keys, all-uncompressed ring-only case. family_registry suite: 34 passed; planner pipeline: 53 passed.
Four of Florin Braescu's seven patches from #1032, rebased onto dev (the route-trace tmp-path and make-install ones are already fixed on dev by other commits, in equivalent or better shape): - quant.h: move g_idot/g_i4s/g_xexp into colibri.c, the only translation unit that uses them — every other engine including quant.h tripped -Wunused-variable - Makefile: 'colibri' was phony while colibri$(EXE) expands to the same name on POSIX, so the alias forced a relink on every invocation; keep the alias only where $(EXE) makes it distinct - test_k3_ram_budget.c: RESERVE now derives from KV_GB instead of duplicating the literal - python tests: close HTTP responses/errors explicitly, and mock physical_cpu_count so the suites stop probing the host All patches by @florin65 (#1032), rebased and verified: zero-warning build on colibri/inkling/kimi_k3/olmoe, make test-c green, python suite green except test_cpu_vs_cpu_determinism, which fails 4/5 runs on clean dev on this host too (wall-clock flake, unrelated).
…ee their buffer coli_cuda_tensor_upload marked ownership only after the weight memcpy succeeded, so a malloc-success/memcpy-failure upload freed the tensor while weights_owned was still 0 and coli_cuda_tensor_free's ownership gate skipped the cudaFree: every failed upload leaked weight_bytes of device memory, invisible to the ledger (tracked=0). Ownership is a fact of the allocation, not the copy — set it between the two. tests/test_weights_owned_cuda.cu pins the contract by including the backend directly and renaming its cudaMemcpy call sites to a hook that fails H2D copies of the target byte count: 8 injected create-fail cycles must leave cudaMemGetInfo free memory unchanged (previously a monotonic ~64 MiB/cycle loss), the success-path control must round-trip to zero delta, and the ledger must never see a failed upload. Wired into cuda-test and gpu-compile alongside the other direct-include tests.
…h's mapped surface The hook named cudaMemcpyKind and cudaErrorInvalidValue, which the compat header does not alias (the backend never uses them), so the cuda-test recipe under HIP=1 failed to compile. Compat's precedent is to map only what the backend needs; alias the kind type and the injected error value locally per vendor instead of widening the product header for a test-only seam. CUDA behavior unchanged.
… the HIP build backend_gpu_compat.h maps only the CUDA runtime names backend_cuda.cu itself uses; the injection hook's kind type and injected error value are not among them, so hipcc -- which compiles this file in the HIP syntax-check lane via gpu-compile -- finds them undefined. Alias the two locally under the HIP arm and keep the shared hook body in the cuda spelling, the arrangement the repo's other direct-include tests use for their off-surface names. The hook's forward declaration also no longer leans on hipcc pre-including the runtime header: the HIP arm now includes <hip/hip_runtime.h> explicitly, the same header backend_gpu_compat.h pulls, unreachable under nvcc. CUDA behavior unchanged.
fix(telemetry): honour USAGE_SAVE=0 in every engine (#1039)
chore: land the v1.6.2 warning-cleanup patches from #1032
fix(CUDA): weights_owned before the H2D copy — rebase of #1108 by @monotophic
fix(cache): make LRU victim selection respect a lowered ecap (#1034)
chore: version 1.7.0
Replace the now-empty unproven-planners refusal test with a positive test asserting every production planner (olmoe, kimi_k3, inkling, deepseek_v4) returns a PlannerGeometry with real resident state -- never a silently-invented zero-byte budget.
…he eviction in datapoint.py
planner: add OLMoE + Kimi K3 + Inkling + DeepSeek V4 geometry adapters (closes #1066)
…ling refactor(serve): migrate Inkling audio framing to shared codec
fix(datapoint.py): add deepseek_v4 CLI support and zero-write fadvise cache eviction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release PR:
dev→main. Merge with--admin(protected branch).80+ pull requests since v1.6.2. Full detail in
CHANGELOG.md; the headlines:A sixth engine, CPU and GPU
qt_ready()gate saves CPU-only builds 7.33 GB.The expert matmul path, rebuilt — every step bit-identical
Layer-level activation hoisting (#1071/#1075/#1076/#1077), the plane-nibble int4 layout + unsigned-VNNI dot (#1079), the 1×4 union tile (#1088), the fused expert pass (#1093) and grouped IDOT for gs64 containers (#1094). Measured 1.45–3.10× on the expert matmul, peak 253 GMAC/s, zero bytes added to any container.
Streaming
DeepSeek V4 loader lanes 3 → 9 (#1097): 1.41× decode on the real V4-Flash checkpoint. DeepGEMM sm120 fetched at a pinned commit instead of vendored (#1056, @dcutugno): 2.5× prefill on sm120, nothing third-party in the tree.
The CI that closes a class of bug
An ARM job plus an integer-kernel bit-exactness gate running on both ISAs (#1083). Every tiny-oracle job ran on x86 before this, which is how IDOT defaults that were not token-equivalent shipped twice — both now fixed (#1044, #1080), along with the ARM64 dotprod probe (#1109, @SebaWag) and a CUDA warp-sync hazard found with
compute-sanitizer(#1111, @monotophic).Contributors
@kreuzzelg · @monotophic · @terrizoaguimor · @ZacharyZcR · @dcutugno · @RDouglasSharp · @bherald · @SebaWag · @outtodata · @lineape · @aaristov · @benmaster82 · @zh-Processor · @8PotatoChip8 · @weber-software — and everyone who filed a datapoint or a bug this cycle.