fix(cuda): charge the expert tier real VRAM, not logical bytes (#687)

[Unknown-Findout]

Closes the sizing half of #687.

@terrizoaguimor measured an unaccounted 0.741 MiB per resident expert (sigma 0.019, five clean configurations, H100 and H200) and concluded it was a fixed workspace cost the 2 GB reserve does not cover, adding that 0.741 is "this model on this card and not a constant worth freezing into the source" and that it should be measured live from the first uploads instead.

The distrust of the constant was right. The reason was not, and that makes the fix simpler than proposed.

What the term actually is

cudaMalloc rounds a request up, and nothing charged the difference. Measured on an RTX 3090 (sm_86, CUDA 13.1):

request              actual      overhead
0.75 MiB          1.00 MiB        +0.25     <- int4-g64 scale array
1.00 MiB          1.00 MiB        +0.00
1.00 MiB + 1 B    2.00 MiB        +1.00
1.50 MiB          2.00 MiB        +0.50
2.00 MiB          2.00 MiB        +0.00
2.00 MiB + 1 B    4.00 MiB        +2.00
6.00 MiB          6.00 MiB        +0.00     <- int4-g64 weight tensor

A GLM-5.2 int4-g64 expert is six allocations of two sizes: three 6 MiB weight arrays, already on a boundary and costing nothing extra, and three 0.75 MiB scale arrays, each padded to 1 MiB.

3 x 0.25 MiB = 0.750 MiB per expert

Against 0.741 sigma 0.019, so inside the interval, from an independent direction and a different architecture.

So the term is roundup(scale_bytes) - scale_bytes. It is a property of the allocator and the model geometry, knowable before a single expert is placed - no fixed-point iteration, and no measurement pass over real uploads. It also predicts what a different group size or hidden dim would cost, which a fitted constant cannot.

The change

coli_cuda_alloc_footprint(bytes) probes the real amortised cost, once per distinct size, cached. Probed rather than modelled on purpose - the table above is not a rule this hardcodes, since the rounding is a driver and architecture property and a formula fitted to one card would be silently wrong on the next.

coli_cuda_tensor_vram(t) applies it per allocation. The weights and the scales are separate cudaMallocs and each is rounded on its own; summing first and rounding once would miss the scale array's padding entirely, which is the whole term.

The tier now decrements remaining by VRAM.

Two byte counts stay logical, and that separation is the design:

• coli_cuda_tensor_bytes() is the LOGICAL size and remains so. Its own comment requires it to mirror upload and free exactly so the three cannot drift, and a test pins that. VRAM is a different question, so it gets a different function rather than a changed meaning for this one.
• m->gpu_expert_bytes is also logical. It is what gets reported as the tier's size, and quoting allocator padding to the user as model bytes would trade one wrong number for another. Only remaining moves, because only remaining is a budget.

Two implementations that did not survive measurement

Both are in the source comments so nobody rebuilds them.

cudaMemGetInfo per placed expert is the obvious exact answer. It is O(live allocations):

live allocations   us/call
0                    0.52
1,000                2.64
5,000               10.80
12,000              68.20

A tier placing 6,235 experts holds 18,705 allocations, so charging it per expert is quadratic.

A single-allocation probe over-reports by 3x. One cudaMalloc reserves a whole 2 MiB VMM page, so one allocation of anything smaller reads as a flat 2 MiB:

786,432 B, n=1    2.00 MiB   <- the page, not the allocation
786,432 B, n=64   1.00 MiB   <- the amortised truth

That version shipped in my first draft and would have shrunk the tier worse than the bug it fixes. Its own test caught it, which is the reason the negative controls below are there.

Verification

Eight assertions in tests/test_alloc_footprint_cuda.cu, against the live allocator rather than a table. Two are negative controls:

• an aligned 6 MiB request must be charged exactly 6 MiB. Without this the suite passes on a function that just doubles everything - exactly what the first draft did.
• 500 repeat calls must move free VRAM by 0 bytes, checked by reading free VRAM rather than by timing.

The probe is also cross-checked against an independent bulk measurement in the same run, so the function and its ground truth cannot drift together.

Wired in at the end of cuda-test: it is the only test there that allocates in bulk to measure an amortised cost, so it should not leave the card fragmented underneath a kernel test that follows it.

Full make cuda-test on sm_86, MAKE_EXIT=0, all eight:

backend_cuda        q8/q4/q2/f32/e8 correctness ok on 1 device(s)
ragged_attention    ok
fp8_warp            shared-lut 0, hw-cvt 0, mutated entry 1 mismatches (fires)
absorb_determinism  batch_differing=0/145  ragged_differing=0/29
fp8_cuda            oracle 0 mismatches, API 0 mismatches
weights_owned       8 fail cycles, 0 bytes cumulative
mxfp4               ok
alloc_footprint     ok, per-expert unaccounted 0.7500 MiB

Python suite Ran 696 tests, OK, skipped=52. CPU-only build compiles clean, so the non-CUDA path is unaffected.

What this does not do

It does not touch the evict-and-retry half (#696), and it does not explain why a rescued tensor computes a different answer than both the healthy GPU run and the CPU fallback. That is a separate defect and this does not go near it.

Tested on Windows 11, RTX 3090 (sm_86), CUDA 13.1 V13.1.115, MSVC 19.50.35725, GNU Make 4.4.1.