v1.5.0 leaves ~60 GB of RAM unused and loses 13 points of expert hit rate: 18% slower decode than v1.4.0 on a 128 GB single-GPU host

[mohamedmastouri2000-boop]

v1.5.0 is 17–19% slower at decode than v1.4.0 on this box, on both an int4-g64 and an E8/IQ3 container. Same machine, same hour, same frozen usage history, two replicates per arm on each version.

The interesting part is not the throughput, it is the resident set. v1.4.0 uses 106 GB of the 128 GB available. v1.5.0 uses 47 GB and never comes back for the rest. Expert hit rate falls with it, 97.4% → 84.4%. The engine is declining memory it has.

Setup

RTX 5080 / sm_120 / 16 GB · Core Ultra 9 285K · 128 GB DDR5-4400 · SK Hynix PCIe 4.0 NVMe · Windows · GLM-5.2 744B.

Both versions built from source with the same toolchain (nvcc -arch=sm_120, make colibri.exe CUDA_DLL=1 ARCH=native) — not the release zip, which carries no sm_120 coli_cuda.dll.

COLI_CUDA=1 CUDA_DENSE=1 CUDA_EXPERT_GB=4 CUDA_RESERVE_GB=1
PIPE=1 DIRECT=1
CACHE_ROUTE=1 ROUTE_J=3 ROUTE_M=16 ROUTE_ALPHA=0.5
USAGE_SAVE=0
--ngen 64 --temp 0, same prompt throughout

Preconditions recorded in both logs, because my first attempt at this was invalid without them — other GPU tenants had clamped the expert tier and the whole comparison was worthless:

VRAM  16303 MiB total, 1739 MiB used, 14239 MiB free
RAM   127.6 GB total / 107.6 GB free

.coli_usage is restored from a master copy before every single run and USAGE_SAVE=0, so the history is identical for all fourteen runs and no run can mutate it mid-flight. Warm-up discarded on each arm. v1.5.0 ran first, then the v1.4.0 binaries were reinstalled and the identical script re-run — same session, nothing else changed.

Result

arm	v1.4.0 b085b48	v1.5.0 8f512fc	delta
int4-g64 decode	1.21, 1.23 → 1.22	1.00, 1.01 → 1.005	−17.6%
E8/IQ3 decode	1.58, 1.58 → 1.58	1.27, 1.28 → 1.275	−19.3%
E8 speed profile J2/M24	1.66	1.46	−12.0%
g64 prefill, 19 tok	17.2, 17.8 s	22.6, 22.1 s	+27%
E8 prefill, 19 tok	10.1, 10.0 s	11.4, 11.4 s	+13%

Replicate spread is ≤0.02 tok/s on every arm on both versions, so this is well outside noise.

Where it comes from: the resident set, not the codec

	v1.4.0	v1.5.0
g64 hit rate	93.3 / 93.5%	84.3 / 84.9%
E8 hit rate	97.4%	84.3 / 84.4%
E8 speed-profile hit	99.1%	90.9%
g64 RSS	105.8 GB	64.1 GB
E8 RSS	106.5 GB	47.5 GB
E8 pin / lru split	72.9% / 24.5%	57.9% / 26.4%

E8 loses 59 GB of resident set. The pinned share drops ~15 points while the LRU share gains only ~2, so the difference is not moved between tiers — it is simply not allocated. Hit rate follows, and throughput follows hit rate.

The two containers are unchanged files, the CUDA expert tier is the same size in both versions (156/1352 for g64, 228/1352 for E8, 3.3 GB), and the card is quiet. Nothing but the engine version differs.

Suspect

#815, perf(cache): preserve adaptive LRU during autopin (@bherald), which adds:

/* Automatic history pinning and the adaptive LRU share the expert RAM budget.
 * Preserve the LRU capacity affordable before pinning, up to the requested ... */
static double autopin_preserve_lru(double planned_pin, double expert_available,
                                   double lru_reserve){
    if(planned_pin<=0.0 || expert_available<=lru_reserve) return 0.0;
    double max_pin=expert_available-lru_reserve;

The intent is right — autopin was starving the LRU. But on this host the reserve appears to be computed far larger than what the LRU then uses, so max_pin is cut hard and the reserved remainder is never claimed by either tier. Both tiers end up small and ~60 GB sits idle. That matches the measured shape exactly: pin down 15 points, LRU up 2, total down 13.

I have not read enough of autopin_lru_reserve() to say which term is wrong, and I would rather hand over the measurement than a diagnosis I have not earned.

My .coli_usage carries 6,592,176 selections, which is large. If the reserve scales with history size this would show up more on well-used profiles than on fresh ones. That is a guess, and it is testable: the same sweep on a fresh .coli_usage would separate it.

What would settle it

1. Print the computed lru_reserve and the resulting max_pin next to the existing [PIN] placement line. If the reserve reads ~60 GB on a 128 GB host, this is confirmed at a glance and by every user who hits it.
2. The RSS figure already appears in the run summary — the regression is visible there. Anyone with both versions reproduces it in two runs.

Happy to re-run any arm, sweep PIN_GB, test a candidate patch, or run with a fresh usage history. Raw logs for all fourteen runs available.

Note for anyone reading this before it is fixed: v1.5.0 also carries eight security advisories, two of them in the model loader. This is a performance regression, not a reason to stay on v1.4.0 if you load models you did not build yourself.

[JustVugg]

I made some adjustments for this case and merged in the dev you have the possibility to try and see if it turns out better.
@mohamedmastouri2000-boop

[MasterCATZ]

I had to do some tweaks to get it using more than 100gb of my 512gb RAM

[JustVugg]

This is the most useful regression report this project has received. Fourteen runs, a frozen .coli_usage restored before each one and USAGE_SAVE=0 so no run can mutate it mid-flight, preconditions recorded because your first attempt was invalid without them, replicate spread ≤0.02 tok/s, and a hypothesis you explicitly declined to over-claim. Thank you.

I believe this is already fixed on dev, and it is the same defect as #856 reaching you by a second path. Analysis below, because you have earned an argument rather than an assurance.

Your suspect is right about the mechanism and wrong about the cause

autopin_preserve_lru() is not miscomputing anything. It is being handed an inflated lru_reserve:

lru_reserve = autopin_lru_reserve(expert_available,
                                  expert_cache_bytes_per_slot(&m, ebits),   /* <-- here */
                                  m.ecap, &preserved_cap);
pin_bytes   = autopin_preserve_lru(planned_pin, expert_available, lru_reserve);
                                   /* max_pin = expert_available - lru_reserve */

In v1.5.0, expert_cache_bytes_per_slot() returned nsp × the widest width in the container. #793 changed expert_bytes_probe() to return the container maximum — correct for the shared ws[64] working set, which really is reused across rows, and wrong for the per-layer LRU caches, each of which holds only its own row's experts.

So on any container that mixes widths — GLM-5.2 ships int4 or E8 routed experts with an int8 MTP head — every row is priced as int8. The reserve is inflated, max_pin is cut by exactly that inflation, the LRU then fills only to its real size, and the reserved surplus is claimed by neither tier.

Which is the shape you measured and could not explain: pin down ~15 points, LRU up ~2, ~60 GB idle.

The arithmetic, against your two containers

One GLM-5.2 expert is 3 × 2048 × 6144 = 37.75M weights:

E8/IQ3    3.06 bit  -> 14.4 MB        int8 MTP  8.01 bit -> 37.8 MB
int4-g64  4.50 bit  -> 21.2 MB

             bytes_per_slot     v1.5.0     dev      inflation
E8/IQ3       77 rows x widest   2.91 GB   1.12 GB     2.60x
int4-g64     77 rows x widest   2.91 GB   1.63 GB     1.79x

At expert_available ≈ 100 GB, the pin budget the two versions permit:

E8/IQ3     cap 25: max_pin  27.2 -> 72.0 GB   (+45 GB)
           cap 30: max_pin  12.7 -> 66.4 GB   (+54 GB)
           cap 35: max_pin   0.0 -> 60.8 GB   (+61 GB)

int4-g64   cap 25: max_pin  27.2 -> 59.2 GB   (+32 GB)
           cap 30: max_pin  12.7 -> 51.1 GB   (+38 GB)
           cap 35: max_pin   0.0 -> 42.9 GB   (+43 GB)

Against your measurements:

	you measured	predicted
E8 resident lost	59 GB	45–61 GB
g64 resident lost	42 GB	32–43 GB
E8 worse than g64?	yes	yes — 2.60x vs 1.79x inflation

The third row is the one that persuades me. E8 loses more than g64 because the ratio between the int8 MTP head and the routed width is larger — 2.60 against 1.79. That is a prediction the model makes before looking at your numbers, not a curve fitted to them afterwards.

Why v1.5.0 and not earlier

#793 introduced the wrong width. #815 — the PR you suspected — made it load-bearing: before it, nothing subtracted the LRU reserve from the pin budget, so an inflated reserve was harmless. @bherald's change is correct and did not cause this; it exposed a miscalculation that was already there. Both landed in v1.5.0, which is why the interaction appears exactly at that tag.

What is on dev

#869 replaced the divisor with the sum of each row's real width, in cap_for_ram() and in expert_cache_bytes_per_slot() — the function feeding your reserve. It also stops wide slabs migrating into narrow rows through the ws[]↔LRU swap, which is what made the pessimism defensible in the first place.

The test asserts magnitude under a controlled A/B on GLM-5.2's real geometry: adding the int8 MTP row costs 3.6% of the cap after, and 51.5% before.

What would settle it, and only you can

Your sweep, unchanged, against dev. Same script, same frozen .coli_usage, same preconditions. What I expect:

• E8 RSS back to ~106 GB, g64 likewise
• hit rate back to ~97% on E8
• decode back to ~1.58 / ~1.22

If RSS comes back and throughput does not, then there is a second regression underneath this one and your setup is the only place we would see it separated. If RSS does not come back, my analysis is wrong and I would rather know that than have it assumed.

Your two suggestions are both going in regardless — printing lru_reserve and the resulting max_pin next to [PIN] placement is exactly right, and is the kind of thing that would have made this visible without fourteen runs. That is a separate PR and I will link it here.

One more thing worth saying: your note at the bottom, telling readers not to stay on v1.4.0 over this because v1.5.0 carries eight advisories including two in the model loader — that was the right call and not everyone would have made it.

[JustVugg]

Closing as fixed on dev by #869 — the analysis is in the comment above.

Short version: your suspect (#815) was right about the mechanism and the cause is one level up. autopin_preserve_lru() computes correctly; it is handed an lru_reserve inflated 2.60x on your E8 container and 1.79x on g64, because expert_cache_bytes_per_slot() priced every row at the container's widest width. The reserve is subtracted from the pin budget, so the pin starves, the LRU fills only to its real size, and the surplus is claimed by neither — your ~60 GB.

The predicted recovery matched your measurements on three independent points, including that E8 loses more than g64 because its inflation ratio is larger.

Please reopen if your sweep on dev does not restore it. Your setup is the best instrument this project has — fourteen runs, frozen .coli_usage, ≤0.02 tok/s spread — so a null result from you is worth more than a pass from anyone else. What I expect: E8 RSS back to ~106 GB, hit rate back to ~97%, decode back to ~1.58.

Your two suggestions — printing lru_reserve and the resulting max_pin next to [PIN] placement — are still wanted and will follow separately. That number would have made this visible without fourteen runs, which is exactly the lesson.

[mohamedmastouri2000-boop]

Ran it. dev @0f4598b, same script, same frozen .coli_usage, same preconditions. The result splits cleanly: the cap is fully restored, the resident set is not, and the two containers now fail in different ways.

Your prediction was RSS back to ~106 GB, hit ~97%, decode ~1.58. That did not happen — roughly half of each came back. You asked to be told either way, so here it is in full, including one number I cannot yet explain.

Setup

Identical to the original report: same box, same two containers, USAGE_SAVE=0 plus master-copy restore before every run, warm-up discarded. Built from source, nvcc -arch=sm_120. #869 confirmed present:

0f4598b Merge pull request #869 from JustVugg/fix/cap-mixed-width
7c97625 plan: price each expert row at its own width, not the container's widest (#856)

VRAM  16303 MiB total, 2105 MiB used, 13873 MiB free
RAM   127.6 GB total / 97.5 GB free

The engine's own budget probe read [RAM_GB=120.9] in all three sessions — v1.4.0, v1.5.0 and dev. The machine state is comparable across the whole comparison, not merely similar.

The cap fix works exactly as designed

	v1.4.0	v1.5.0	dev @0f4598b
g64 cap raised 8->	31	17	31
E8 cap raised 8->	46	17	45

That is your arithmetic landing. The width pricing was the defect and #869 fixes it.

The engine does not fill the cap it now permits

	v1.4.0	v1.5.0	dev
g64 decode	1.21, 1.23	1.00, 1.01	1.12, 0.98, 1.11, 0.97, 1.05
E8 decode	1.58, 1.58	1.27, 1.28	1.43, 1.43
E8 speed J2/M24	1.66	1.46	1.53
g64 hit	93.3 / 93.5%	84.3 / 84.9%	88.9-89.4%
E8 hit	97.4%	84.3 / 84.4%	91.8-92.1%
g64 RSS	105.8 GB	64.1 GB	84.67 GB (all 5 runs)
E8 RSS	106.5 GB	47.5 GB	75.6 GB

A budget that permits ~106 GB is being filled to ~76-85.

The two containers now fail differently, and that is the useful part

Recovery as a fraction of the v1.4.0 -> v1.5.0 loss:

	RSS	hit	decode
E8	48%	58%	51%
g64	49%	51%	19%

On E8, throughput tracks residency — half the memory back, half the speed back. On g64 half the residency came back and the throughput did not follow.

That is the branch you named in advance. It suggests the remainder is not one defect: E8 is still short of memory, while g64 has memory it is failing to convert.

An anomaly I am reporting rather than explaining

g64 on dev does not replicate. Five runs: 1.12, 0.98, 1.11, 0.97, 1.05 — mean 1.05, spread 0.15, and it reads almost bimodal.

For contrast, every other arm measured on this box today:

v1.4.0 g64   1.21, 1.23      spread 0.02
v1.5.0 g64   1.00, 1.01      spread 0.01
dev    E8    1.43, 1.43      spread 0.00
dev    g64   1.12 .. 0.97    spread 0.15

Across all five g64 runs the hit rate held at 88.9-89.4% and RSS was identical to the byte — 84.67 GB every time. So residency is deterministic and decode is not. Something downstream of placement is varying run to run, only on dev, only on the mixed-width container that also carries the int8 MTP head.

I do not know what that is. I am flagging it because a bimodal decode on one build and one container is the kind of thing that gets averaged away and then rediscovered as a mystery three releases later.

Where the counters point on the remaining half

E8	pin	lru	total
v1.4.0	72.9%	24.5%	97.4%
v1.5.0	58.0%	26.4%	84.4%
dev	51.7%	40.4%	92.0%

The LRU has expanded into the freed reserve — 24.5% -> 40.4% — which is #869 working. The pinned share has not recovered; it is lower than on v1.5.0 and 21 points below v1.4.0.

So the reserve is no longer inflated and the LRU is now claiming it. What has not returned is the pinned set. That reads like a constraint on the pin path independent of lru_reserve — but I guessed wrong once in this thread already and I am not going to do it twice in public.

What I can run next

A PIN_GB sweep on dev; a PIN=off run to isolate the LRU ceiling alone; a fresh-.coli_usage run to test whether any of this scales with my 6.59M-selection history; or more g64 replicates to characterise the bimodality. Say which is most useful.

The lru_reserve / max_pin logging you agreed to add would answer the pin question directly — with those two numbers beside [PIN] placement I could tell you in one run whether the pin path is clamped or simply declining to grow.

On the headline: #869 is a real fix and should ship. The mechanism you identified is confirmed by the cap numbers, and roughly half the regression is gone. I would not close this on it — a 128 GB host still leaves ~30 GB unused, g64 still cannot convert what it did get back, and #856 presumably sits on the same remainder.

Raw logs for all nineteen runs across the three builds available on request.

[JustVugg]

This is the follow-up I asked for and it says my fix was partial. Taking that plainly first: the prediction was RSS back to ~106 GB and it did not happen. You measured ~76-85. I was wrong about sufficiency, and your split of the result is what makes the remainder tractable.

What landed

g64  cap raised 8->31   (v1.4.0: 31, v1.5.0: 17)
E8   cap raised 8->45   (v1.4.0: 46, v1.5.0: 17)

The width pricing is fixed. [RAM_GB=120.9] identical across all three sessions removes the obvious confound, and you did that before I could ask.

Why the cap came back and the residency did not

I found the second constraint, and it is one I did not change. In v1.4.0:

double pin_gb = expert_avail(...) * 0.5 * conf / 1e9;
pin_load(&m, g_usage_path, pin_gb, 0);

That number went straight to the pin. On dev, the same figure is then cut again:

double planned_pin = expert_available * 0.5 * conf;
double lru_reserve = autopin_lru_reserve(expert_available,
                         expert_cache_bytes_per_slot(&m, ebits), m.ecap, &preserved_cap);
double pin_bytes   = autopin_preserve_lru(planned_pin, expert_available, lru_reserve);
                     /* max_pin = expert_available - lru_reserve */

#869 corrected the size of lru_reserve. It did not remove the subtraction. #815 introduced that subtraction and v1.4.0 has no equivalent, so the pin is still bounded by something that did not bound it before — now by the right amount rather than a 2.6x-inflated one, which is exactly half the loss you measured coming back.

And the LRU cannot claim the difference, because it is demand-filled: cap is permitted capacity, allocated on miss. If a turn does not touch cap × rows distinct experts, the remainder is never allocated by anyone. Your v1.4.0 pin/lru split was 72.9%/24.5% — the pin was doing the work, and it is the pin that is still constrained.

Please post the [PIN] lines and the pin/lru split from a dev run. You reported that split for v1.4.0 and v1.5.0 but not for dev, and it is the number that confirms or kills this. If [PIN] auto: X GB plan capped to Y GB to preserve the LRU appears, Y is the answer.

Your E8-vs-g64 split is the more interesting finding

	RSS	hit	decode
E8	48%	58%	51%
g64	49%	51%	19%

Half the memory back and half the speed back on E8 is a proportional response. g64 getting the memory and not the speed is not, and you are right that it points at a different mechanism.

The g64 variance is probably mine, and I flagged it in advance

v1.4.0 g64   1.21, 1.23        spread 0.02
v1.5.0 g64   1.00, 1.01        spread 0.01
dev    E8    1.43, 1.43        spread 0.00
dev    g64   1.12 .. 0.97      spread 0.15   bimodal

Everything on that box replicates to ≤0.02 except the one arm that is new. From #869's own text:

NOT MEASURED, and I would rather say so. The shrink is new work: a slot alternating between the MTP row and main rows now reallocates instead of staying wide, and at ~19-38 MB glibc serves that with mmap/munmap plus the page faults on first touch.

That is a per-run allocator effect, which is what bimodal-with-large-spread looks like. And it should hit g64 harder than E8: the shrink fires on the width gap, and g64's routed rows are 21.2 MB against E8's 14.4, so each reallocation costs more pages while the int8 MTP row is the same size in both.

I would rather test that than argue it. I will add an env switch that disables the shrink at runtime so you can A/B it without a rebuild — if dev with the shrink off replicates to ≤0.02 on g64, the mechanism is confirmed and the fix is to raise the hysteresis or shrink only on promotion rather than on every load.

Where this leaves it

Reopening. #869 fixed one of two constraints; the pin subtraction is the other, and the g64 variance is a third thing I introduced. Three separate items, and your measurement is what separated them — I would have merged the cap fix and called it done.

The [PIN] split from dev is the one thing I need from you before touching anything else.

[mohamedmastouri2000-boop]

Here are the [PIN] lines. They point somewhere other than the subtraction — on dev the pin is byte-identical to v1.5.0, so #869 did not change it at all.

v1.4.0   [PIN] placement: 156 VRAM + 2275 RAM expert (48.3 GB warm)   g64
         [PIN] placement: 228 VRAM + 3360 RAM expert (48.6 GB warm)   E8

v1.5.0   [PIN] placement: 156 VRAM + 1196 RAM expert (45.2 GB warm)   g64
         [PIN] placement: 228 VRAM + 1124 RAM expert (42.5 GB warm)   E8

dev      [PIN] placement: 156 VRAM + 1196 RAM expert (45.2 GB warm)   g64
         [PIN] placement: 228 VRAM + 1124 RAM expert (42.5 GB warm)   E8

Same counts, same warm GB, both containers, all runs. No [PIN] auto: … capped to … to preserve the LRU line appears in any of the three builds — so if lru_reserve were the binding term, correcting it by 2.60x and 1.79x would have moved the pin. It moved by zero.

The warm GB is roughly right. The expert count is half.

Divide one by the other:

g64  v1.4.0   48.3 GB / 2275 = 21.2 MB per expert     <- int4-g64 width, correct
g64  dev      45.2 GB / 1196 = 37.8 MB per expert
E8   v1.4.0   48.6 GB / 3360 = 14.5 MB per expert     <- E8/IQ3 width, correct
E8   dev      42.5 GB / 1124 = 37.8 MB per expert

Both containers land on 37.8 MB — the int8 MTP width, the same figure @ThefloorMiner derived on #766 and you used in your own arithmetic above.

So the pin is filling a roughly correct GB budget while pricing every pinned expert at the container's widest row. It buys half as many experts as the budget affords. That is #793's defect again, in a call site #869 did not reach: you fixed cap_for_ram() and expert_cache_bytes_per_slot(), and whatever converts the pin's GB budget into an expert count still uses the max width.

It also reproduces your own ratio prediction, one level down. E8 is charged 37.8 against a true 14.5 (2.60x) and g64 37.8 against 21.2 (1.79x) — the same two numbers, now visible in the pin count rather than the reserve.

And it explains the split that puzzled us both. E8 is short of memory: pinning 1124 experts instead of ~3360 leaves the residency it never got. g64 got its memory back and cannot convert it — 84.67 GB resident on every run, byte-identical across five, while decode scatters. Those are different problems because the pin miscount hits E8 harder in experts even though both are charged the same per-slot price.

On the pin/lru split for dev

It was in my previous comment, in the last table — dev reads pin 51.7% / lru 40.4% against v1.4.0's 72.9% / 24.5%. Repeating it here since it is easy to miss, and it now reads differently in light of the above: the pin's share fell because it holds half the experts, not because it was clamped.

Your allocator hypothesis for the g64 variance

That fits what I see and I would like to test it. Send the env switch whenever it is ready and I will run g64 five times with the shrink on and five with it off, same frozen .coli_usage, same preconditions. If it replicates to <=0.02 with the shrink off, you have it.

One detail that may support you before the test: RSS is identical to the byte across all five g64 runs on dev — 84.67 GB — while decode ranges 0.97 to 1.12. Whatever varies costs time without changing the footprint, which is what mmap/munmap churn plus first-touch faults would look like.

What I would check first

Whether the pin path derives its expert count from expert_cache_bytes_per_slot() or a sibling that still returns the container maximum. If it does, this is a one-line change of the same kind as #869 and should restore the remaining ~30 GB on both containers.

Happy to run the sweep again the moment there is something to test.

[mohamedmastouri2000-boop]

Built diag/pin-budget and ran it. The new line settles the open half in one load, and the A/B refutes the allocator hypothesis. Both results below, including the one that goes against you.

The placement terms

g64  [PIN] auto: budget 102.2 GB | plan 51.1 GB (conf 1.00, 6592176 selections)
                 | LRU reserve 13.0 GB (cap 8/layer x 1630.3 MB/row-set)
                 | max_pin 89.2 GB -> pinning 51.1 GB

E8   [PIN] auto: budget 102.2 GB | plan 51.1 GB (conf 1.00, 6592176 selections)
                 | LRU reserve  9.0 GB (cap 8/layer x 1121.6 MB/row-set)
                 | max_pin 93.2 GB -> pinning 51.1 GB

No [CAPPED by the LRU reserve] on either container. max_pin is 89.2 and 93.2 GB against a plan of 51.1. The reserve is nowhere near binding, so the subtraction #815 introduced is not the remaining constraint. That is the hypothesis dead, and the line you added is what killed it — it took one 8-token load instead of fourteen runs.

#869 is also confirmed correct here. The row-set widths are right: 1630.3 / 77 rows = 21.2 MB for g64 and 1121.6 / 77 = 14.6 MB for E8, each container's own width. My earlier framing — that the reserve was priced at the max width — was wrong, and this is what showed me.

Where the loss actually is

Both containers pin exactly 1352 experts:

dev     g64  156 VRAM + 1196 RAM = 1352
dev     E8   228 VRAM + 1124 RAM = 1352
v1.4.0  g64  156 VRAM + 2275 RAM = 2431
v1.4.0  E8   228 VRAM + 3360 RAM = 3588

Two containers whose expert widths differ by 45% cannot land on the same count unless the divisor is container-independent:

51.1 GB / 37.8 MB = 1352    <- dev, both containers, exact
51.1 GB / 21.2 MB = 2412    <- v1.4.0 g64 pinned 2431
51.1 GB / 14.6 MB = 3510    <- v1.4.0 E8  pinned 3588

The reported warm GB agrees: 1196 x 37.8 MB = 45.2 GB and 2275 x 21.2 MB = 48.3 GB, which are the figures in the [PIN] placement lines.

So the plan is right, the reserve is right, the clamp never fires — and the conversion from 51.1 GB of plan into an expert count still divides by the container's widest row. v1.4.0 divided by each container's own width. That is the same defect as #793 in the one place #869 did not reach, and it accounts for the whole remaining gap on both containers.

One thing worth separating: plan 51.1 GB is exactly half of budget 102.2 — the expert_available * 0.5 * conf line, which v1.4.0 had too. Fixing the divisor restores v1.4.0 behaviour; it does not go past it. The other ~51 GB is a separate design question, not a regression.

The slab-shrink A/B: your hypothesis does not survive

Ten g64 runs, interleaved so drift hits both arms equally, frozen .coli_usage, USAGE_SAVE=0:

SHRINK ON    1.13  1.11  1.12  1.10  1.07      mean 1.106   spread 0.06
SHRINK OFF   1.10  1.11  1.12  1.12  0.96      mean 1.082   spread 0.16

Turning the shrink off did not tighten anything. The single low run moved to the off arm. If reallocation were the cause, off should have been the clean arm and it was the scattered one.

Across all fifteen g64 runs I now have on this build, thirteen sit in 1.05-1.13 and three sit at 0.96-0.98, and the lows appear with the shrink on and off. That reads like an occasional external event on this box — roughly one run in five, costing ~15% — rather than anything #869 introduced. RSS was 84.67 GB on all ten runs, and hit rate 88.6-89.7%, so whatever it is does not touch placement.

I would drop the allocator as the explanation and treat the g64 variance as unexplained-but-probably-mine. It should not hold up the pin fix.

Net

• #899 should merge on the strength of the first section alone: it converted a fourteen-run measurement into a one-load read, and immediately falsified the leading hypothesis.
• The remaining half of #885 is the pin's GB-to-count divisor, and the [PIN] auto: line now makes that checkable by anyone in one run.
• COLI_SLAB_SHRINK earns its place as a diagnostic even though it came back negative — a null result this cheap to obtain is worth having.

Happy to re-run the full sweep the moment the divisor changes. If it is the one-liner it looks like, I would expect g64 back to ~2400 pinned / ~105 GB RSS / ~1.22 and E8 to ~3500 / ~106 GB / ~1.58.

[JustVugg]

#899 is on dev (47a749d) — please re-run when you have a moment. It does not attempt the remaining fix; it makes the remaining fix decidable, which is what your last set of numbers showed was missing.

The line you asked for, twice

[PIN] auto: used to print only when the clamp bit, which made "the reserve was zero" and "the reserve did not bind" indistinguishable from outside — the exact two hypotheses your data leaves open. It now prints every term, every run:

[PIN] auto: budget X GB | plan Y GB (conf C, N selections) |
      LRU reserve Z GB (cap K/layer x W MB/row-set) |
      max_pin M GB -> pinning P GB  [CAPPED by the LRU reserve]

One run of each container is enough. If LRU reserve sits near budget, the pin is starved by a reservation the demand-filled LRU may never claim, and that is the second constraint I described. If it does not, my reading is wrong and the remaining ~30 GB is somewhere else entirely.

I could not derive this from here: your v1.4.0 split of 72.9% pin on 106.5 GB implies ~77.6 GB pinned, but planned_pin = 0.5 * expert_available would need a ~155 GB budget on a 128 GB host. Something in my model of expert_avail() is wrong, and rather than guess a third time I would like the engine to say.

And the switch for the g64 variance

COLI_SLAB_SHRINK=0    # restores grow-only, the pre-#869 allocator behaviour

Same binary, no rebuild. If g64 replicates to ≤0.02 with it set and stays at 0.15 without, the bimodal spread is the slab shrink I introduced and flagged as unmeasured — and the fix is to raise the hysteresis or shrink only on promotion rather than on every load.

It re-introduces the slab migration, so the per-row cap becomes optimistic again. Diagnostic, not a configuration — the engine says so on stderr when it is set.

One thing that will look different

test_cap_mixed_width gained a case because the first version of that switch proved nothing: the test drives expert_load directly and never runs main(), where the environment is read, so it passed identically with the shrink on and off. Mentioning it because you are the person most likely to check whether a knob does anything before trusting it.

Three separate items now: the width pricing (fixed), the pin subtraction (identified, unfixed), and the shrink variance (mine, now isolatable). Your measurement is what separated them — I would have merged the cap fix and called it done.