cuda: zero-copy expert views on pageable-shared memory (GB10)

[Nanetnounou]

What

On integrated devices with pageableMemoryAccess=1 (GB10 / DGX Spark class: one LPDDR5X pool shared by CPU and GPU), uploading an expert copies bytes the GPU can already read in place — and with CUDA_RELEASE_HOST each device copy also keeps a mirror host page, so the same physical bytes are paid twice. Worse: experts that are not resident on the device tier fall back to CPU compute, a rational choice on a discrete GPU and a pure loss on unified memory.

This PR teaches the CUDA backend non-owning device views onto host slabs (coli_cuda_tensor_wrap, using the weights_owned field already present in dev):

• streamed experts and PILOT-prefetched experts (demand=0) compute on the GPU through views — without the demand=0 case, PILOT resurrected 914 expert rows on the CPU per decode window and gave back its I/O gain; measured after: routed CPU rows during decode = 0;
• pinned experts are left without device copies after placement (a view suffices; the pass must run after the RAM-tier load loop and after placement — both other orderings were measured wrong);
• the io_uring loader wires the same view path (it set the format but not the view, so uring loads silently fell back to copies);
• fix in coli_cuda_tensor_upload: a view must never satisfy a request for a copy — the cached-copy shortcut matches on fmt/I/O/device, would accept a view, and the view's weight_bytes=0 then collapses expert-tier accounting (measured: tier fell from 3795 to 128 experts). Latent on dev today, armed the moment anything creates views.

Together with the E8 warp kernels (separate PR), decode p50 on GLM-5.2 E8-IQ3 on GB10 went 1376.6 → 640.9 ms/token (2.15x).

Known limitation (deliberate)

COLI_GROUP_ASYNC=1 must stay off while views are in use: the ESlot LRU has no reference count, so the async group path can recycle a slot while a GPU read is in flight. Reported with full analysis in #933; the sync path costs ~3.5% of orchestration time. A per-slot refcount is the real fix and is out of scope here.

Dropped during rebase

An earlier version also changed GPU-tier capacity planning to use routed-expert width; dev's expert_bytes_row (#856) already solves that better, so it was dropped.

Suggestion (no code in this PR)

On devices with pageableMemoryAccess=1, the unified behavior could become the default: views on, GPU prefill pipeline on, async group path guarded off. Today that requires a combination of env vars that only someone who has done this debugging would know. Happy to send a follow-up PR if there is interest.

Tests

New c/tests/test_zerocopy.c. make check: 415 tests OK. Same note as the kernels PR: cuda-test exits 1 on GB10 already on a pristine dev checkout (v1.5.0-based builds pass on the same machine/driver) — separate issue to follow.

Disclosure

Developed with AI assistance (Anthropic Claude), human-reviewed; all figures were measured on the hardware described.

[JustVugg]

On the trilogy (#934, #935, #936, plus the #933 analysis behind it): this is serious work and it's in the review queue — but deliberately after the imminent release. The release carries the v1.5.0 regression fix and is soaking now; GPU/kernel changes land in the next cycle where they get proper time, and #936 in particular needs its lifetime story reviewed against your own #933 refcount analysis before it can be on by default.

Two things that would speed review: confirm #936 contains (or depends on) the per-slot refcount from #933 rather than preceding it, and — since we have no GB10-class hardware — a second measurement from anyone with unified-memory NVIDIA would help. The NEON path (#934) is the least entangled of the three and will likely go first.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.

Three things — a state update that affects the review question, a technical
question from adjacent work, and the measurement asked for above, which we're
running.

1. The #933 dependency question is now a different question. Since the
comment above, #933 was closed (08-14) by #989's slot-lifetime fix (merged
08-13) — a different implementation from the one this PR was written against,
and this branch hasn't been rebased since. So the lifetime story to review is
no longer "does #936 contain the per-slot refcount" but "how do #936's
zero-copy views interact with the slot-lifetime scheme #989 already landed" —
a view that outlives its slot's recycle window is the #933 hazard reappearing
one level up. We went through #989's ground while validating #1037 and will
cover that seam in our pass.

2. A question from #1037's cycle, registered before we measure: do the
zero-copy views preserve — or assert — the alignment guarantees the copied
path provided? A view pointer inherits the source mapping's alignment, not the
staging copy's, and kernel dispatch that keys on 16/32-byte alignment can
silently select a different path under views than under copies: same math on
paper, different kernels in flight, and any difference shows up as unexplained
numerics or throughput drift rather than an error. If the answer is "pinned by
construction," one assert would document it cheaply.

3. The unified-memory NVIDIA measurement: we're running it. My GB10 pair
(sm_121, CUDA 13, unified LPDDR5X) is the hardware class named above, and
#1037 left us a fresh, reviewed harness on the same ground. The matrix we're
running, so nobody has to wait on a back-and-forth: the branch as submitted
(noting the #989 rebase question), views-vs-copies correctness, decode
throughput at S ∈ {1, 4, 8, 32}, and %-of-wire-speed — normalized against a
measured peak, since on GB10 cudaDevAttrMemoryClockRate returns the
effective transfer rate and a 2×-DDR-derived roofline overstates peak 2×.
It's queued directly behind a measurement pass we're finishing on #935;
results land on this thread. If a different matrix or base would be more
useful for the review, say so and we'll fold it in — but nothing here waits
on that.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic

Here is the second unified-memory measurement asked for above, from my GB10 box (DGX Spark
class, 121 GB unified LPDDR5X, driver 580.173.02, CUDA 13.0, sm_121). We measured the branch
as submitted (f0cef9a) with a standalone harness around the grouped fmt=6 expert path —
same method as our #935 review bench — views (coli_cuda_tensor_wrap) vs copies
(coli_cuda_tensor_upload), 8 experts at GLM decode shapes (D=6144, I=2048), 7 reps,
two independent runs. Full tables and scripts are reproducible from the harness; happy to
attach it if useful.

Correctness. View and copy outputs are bitwise identical at S=1 and S=4 (memcmp over
all outputs, two runs). That is the expected result from the code: dispatch keys on fmt/gs/shape
only and the E8 kernels assemble words byte-wise (e8_ld_u32), so the same kernel reads the
same bytes through either pointer. Double oracle vs the CPU matmul_e8 (real codebook):
maxrel 9.4e-05.

Throughput. Kernel window (COLI_CUDA_PROFILE), stock block kernels, mean±σ over 7 reps:

S	rows	copies (ms)	views (ms)
1	8	5.75 ± 0.24 / 5.61 ± 0.11	5.62 ± 0.01 / 5.61 ± 0.01
4	32	22.25 ± 0.29 / 22.36 ± 0.41	22.31 ± 0.05 / 22.26 ± 0.05
8	64	44.32 ± 0.31 / 44.24 ± 0.20	44.54 ± 0.03 / 44.52 ± 0.10
32	256	176.97 ± 0.46 / 176.50 ± 0.16	178.33 ± 0.43 / 178.26 ± 0.37

(two runs shown as a/b). Views = copies within noise at S≤8, ~1% slower at S=32. Measured
device read ceiling on this box (float4 grid-stride sum, 512 MB, best of 5): 262.7–263.1 GB/s;
both paths run these kernels at ~21 GB/s ≈ 8% of that, i.e. launch/compute-bound — which is
exactly why zero-copy costs no kernel time and why the win lives elsewhere: building the same
24 tensors (115.6 MB) costs 7.1–7.5 ms via upload vs 0.19–0.20 ms via wrap, with zero
device-side bytes (no double residency) and no CPU fallback for non-resident experts. We did
not attempt to reproduce the end-to-end 2.15x p50 figure (this box runs live serving); the
mechanism it rests on checks out.

On the refcount question. #936 as submitted contains no per-slot refcount and does not
depend on one — it predates #989 and instead documents keeping COLI_GROUP_ASYNC=0. It merges
cleanly with current dev (e27c37e7, includes #989); the merged tree builds, and cuda-test
on it fails only at the pre-existing mxfp4 gate (identical failure on pristine dev; likewise
the branch head's cuda-test exit 1 reproduces on its pristine merge-base, confirming the
"pre-existing" note in the PR). Post-merge, #989's in_flight refs and busy-skipping victim
scans (pilot, uring, RSS sweeps) cover view-backed slots on the async path, and the sync group
call completes the stream before returning, so a view's borrow window never exceeds the slab
exposure the CPU path already had. The "async must stay off" limitation looks stale once #989
is in — but we have not runtime-tested COLI_GROUP_ASYNC=1 with views, so treat that as
analysis, not measurement.

Alignment. Nothing in CUDA dispatch keys on 16/32-byte pointer alignment (no
(uintptr_t)&mask tests in backend_cuda.cu); fmt=6 kernels are byte-load by design, and
scales stay 4-byte aligned by construction (fslab float indexing). A view cannot silently
select a different kernel — the bitwise identity above is the empirical confirmation. The one
nit: coli_cuda_tensor_wrap asserts nothing, so a future fmt=0 view would inherit whatever
alignment the slab offset happens to have.

The shipped test needs three small fixes: (1) c/tests/test_zerocopy.c has no Makefile
rule, so make check never builds it; (2) it calls matmul_e8_neon, which doesn't exist —
undefined reference at link (the oracle is matmul_e8); (3) once linked, its
coli_cuda_matmul call routes through coli_cuda_tensor_upload, where this PR's own
view-never-satisfies-a-copy guard silently replaces the view with a real copy before compute —
we instrumented coli_cuda_tensor_owns_weights after the call and it returns 1. So the test
as written validates the upload guard (a real fix this PR introduces), not zero-copy compute.
Asserting through coli_cuda_expert_group instead would test the intended path — that is what
our harness does, and it comes back bitwise-identical.