OLMoE has no hot expert zone (counter-datapoint for pinned hot-store policies)

[outtodata]

TL;DR: measured routing on OLMoE-1B-7B-0924 (520 prompts, 5 domains + Wikipedia en/ru, ~50k tokens): the 32 most popular experts of a layer cover only 75% of routed top-8 slots; globally the top-256 of 1024 (layer, expert) pairs cover 52%. The popularity tail is flat - a pinned/learned hot-store buys much less on this model than a Zipf-heavy one.

Numbers

• Per-layer popularity of the top-32/64 experts: mean 0.752, min 0.681 of slots
• Global (layer, expert) pairs: top-64/1024 = 21.4%, top-128 = 34.7%, top-256 = 51.9%
• Unique experts touched per token: mean 56.3 / 64 (p10 = 54)
• Adjacent-token union Jaccard: 0.83 (temporal locality is high even though popularity is flat)

Method

Router top-8 indices collected per token per layer via HF forward hooks (output_router_logits) on 520 prompts (template domains: code/english/math/prose/russian + Wikipedia summaries en/ru). Collection script and aggregates available; happy to share the npz.

Why it matters

A hot-store sized from popularity curves assumes a steep head. On OLMoE the head is shallow: pinning the top-25% of experts still leaves ~48% of traffic unpinned. Suggest gating hot-store-style policies on a measured popularity skew check (or a flag) for OLMoE-like models. This is meant as a held-out datapoint for the hypothesis table, not a critique - the policy may still win on 256+expert models with steeper routing.

[JustVugg]

Thank you for posting this as a counter-datapoint rather than a bug — it is the more useful shape, and it lands on a policy we do ship.

Your headline is the part that matters to us: top-32 covering 75% of slots, and adjacent-token Jaccard 0.83. Those two together say something specific — that on OLMoE the win is in temporal locality, not popularity locality. Our pinned hot-store is sized from popularity, and PILOT/prefetch exploits the temporal axis. Your numbers say that on this model the second is where the headroom is and the first is close to a waste of RAM.

That is worth acting on, and your suggestion is the right one: gate hot-store policies on a measured skew check rather than assuming a steep head. We already write .coli_usage per model, so the skew is computable from data we have — a model whose top-25% covers only ~52% of traffic should not be spending its budget on pinning.

Two questions, both because I would rather not generalise from one model, which is a mistake we have made here before:

1. Do you have the same curve for a 256-expert model? OLMoE is 64; GLM-5.2 and Kimi K3 are 256 and route top-8. Your hypothesis that the policy may still win there is exactly the thing that needs a datapoint, and if the head is shallow at 256 too, then the hot-store is wrong everywhere and not just on OLMoE.
2. Is the flat head a property of the model or of the prompt mix? 520 prompts across 5 domains plus two Wikipedia languages is deliberately broad. A single-domain workload — someone using this to write C all day — might have a much steeper head, and that is the workload a hot-store is for.

The npz would be welcome. If you would rather not maintain it, the aggregates in the issue are enough to act on the gate.

Related, since you are measuring the same axis: your #866 ceiling number (0.82 recall at h=2 against README's 71.6% for PILOT) is the more consequential finding of the two, because it says our prefetcher is closer to its ceiling than the raw gap suggests — and therefore that effort belongs elsewhere. If those two hold up, together they say: stop tuning the predictor, stop pinning by popularity, and spend it on the read path instead. Which is where #863 is.

[outtodata]

Both answers, honestly:

1. 256-expert model — no, and we can't produce one on this box: 14 GB RAM total, and GLM-5.2/Kimi K3 don't fit in any local form. So no 256-expert popularity curve from us; that datapoint has to come from someone with the RAM for it. If the flat head holds at 256 experts too, your "hot-store is wrong everywhere" conclusion follows — but we'd rather say "unknown" than extrapolate from one model, same as you.

2. Model property vs prompt mix — fair challenge, and we can answer it with the harness we already have: the collect is tagged per-domain (code/english/math/prose/russian + Wikipedia en/ru), so we can recompute the popularity curve on a single-domain subset (code-only, the "writes C all day" workload) without any new inference. We'll post the per-domain top-32/top-256 coverage numbers here. If the head steepens sharply single-domain, the hot-store gating logic should key on the workload skew, not the model skew — which is computable from .coli_usage, as you noted.

npz: the aggregates (per-layer popularity, global (layer, expert) coverage, Jaccard series) are small and self-contained; we'll attach them with the per-domain rerun so the numbers are checkable rather than quoted.

On the broader implication — stop tuning the predictor, stop pinning by popularity, spend it on the read path — our #866 gate-mass numbers (posted today: 0.88 gate-mass recall at h=2) point the same way. And on the read path, note today's #863 comment needs reading before anyone quotes the 3.5x: we failed to reproduce our own slow path and are chasing the confound.

[gcap0n1]

Held-out 256-expert datapoint — the curve asked for in this thread. Computed from a real .coli_usage with the skew check you already described, not a new collector.

What was measured

• DeepSeek-V4-Flash. File header is the documented one: -1 43 256 (layers, experts_per_layer), then -2 1 <engine_id> (skipped). Body is layer expert count.
• 31 841 routed-expert increments (usage[layer*E+expert]++ on each route), 43 layers, 10 634 unique (layer, expert) pairs out of 11 008 possible (96.6 % of the table has been seen at least once).
• ≈741 events/layer. The head (N≤32) is usable; the far tail is thin (~3 counts/pair). This is not a 50 k-token hook collect.
• One 32 GB RAM CPU box, accumulated Colibrì sessions (chat + benches). Not a tagged multi-domain eval. No route-trace, so no adjacent-token Jaccard.
• Per-layer frequency oracle = top-N by count / layer total. That is an upper bound on any static pin of size N. Different unit from the HF-hook “routed top-8 slots” numbers above; same axis.

Per-layer coverage (N / 256)

N	frac of experts	hit_rate
8	3.1 %	0.3557
16	6.25 %	0.4648
32	12.5 %	0.5854 (unweighted mean 0.583, min 0.246, max 0.705)
64	25 %	0.7119 (mean 0.710, min 0.424, max 0.775)
128	50 %	0.8317
192	75 %	0.9243

OLMoE reference in this issue: top-32/64 (50 % of experts/layer) mean 0.752, min 0.681.

Global (layer, expert) pairs (11 008)

top-K	frac of pairs	hit
64	0.6 %	0.1365
128	1.2 %	0.2189
256	2.3 %	0.3204
512	4.7 %	0.4250
2 752	25 %	0.7189
5 504	50 %	0.8389

OLMoE reference: top-256/1024 (25 % of pairs) = 0.519.

Reading (labelled as such)

1. Q1. On this file the 256-expert head is not as flat as OLMoE: 50 % of experts/layer → 0.83 vs 0.75; global top-25 % of pairs → 0.72 vs 0.52. “Hot-store is wrong everywhere” does not follow from this file.
2. A 16-slot-per-layer pin (what this V4 binary used) is 6.25 % of the table and covers 0.46. A 25 %-of-table check calibrated on the OLMoE 0.52 number would classify V4 as steep enough to pin, then spend the budget on 16 slots that miss more than half the traffic. If the gate exists, it has to be against the pin size, not a fixed 25 % of the expert table.
3. Layer variance is large (top-32 min 0.246 vs max 0.705). A uniform pin-N treats a near-uniform layer and a peaked layer the same.
4. Q2. This mix is not a 5-domain eval and the head is still only moderately peaked. A single-domain day might steepen; we do not have a tagged split, so that stays open.

We used 0.60 at N=16 as a local cutoff for raising the pin. It failed (0.4648). 16→24 is +0.069 on this file (0.5342 at N=24). We left the compile-time pin alone.

Oracle is ~40 lines (skip layer < 0, rank per layer). Happy to drop it here if you want it next to a gate.

[lornecodes]

Two things from a different angle on this, one that sharpens your numbers and one that answers the question you close with.

The 75.2% needs a baseline, and it changes what "flat" means

Concentration statistics are biased upward by thin sampling — unobserved experts hold no mass, so the observed ones hold all of it — and more importantly they have no natural zero. "Top half of experts covers 75%" reads as flat only against intuition. Against a uniform random router at the same sampling density it is not flat at all:

top-50% of experts	measured	uniform null	excess
OLMoE-1B-7B, our trace (30 routes/expert/layer)	78.5%	56.8%	+21.7pp

Our 78.5% is close enough to your 75.2% to be the same phenomenon (ours sits slightly higher because at density 30 it is still mildly inflated; yours at ~6000 is essentially converged). But a uniform router already scores ~57% at that fraction. So OLMoE's head is ~20pp above chance — shallower than Zipf, clearly above uniform.

That distinction matters specifically for the thing you propose, "gating hot-store-style policies on a measured popularity skew check": a gate needs a reference point, and without one 75% and 52% are unanchored. We ended up building that check for our own work and it is here — analyze_route_skew.py. It prints sampling density, the measured share, the uniform null at that same density, the excess, and all of it over prefixes so a number that has not converged is visible as one that has not converged. Happy to send it as a PR if it is useful.

We had to write it because we got this wrong first: we reported Qwen3.6 at "top-26% catches 90.0% of routes, the most skewed model we have measured" before checking that a uniform router at our trace's density scores 50.0% on that same measure. Most of the headline was trace length.

"The policy may still win on 256+expert models with steeper routing" — we tested that, and it does not

Qwen3.6-35B-A3B is exactly that case: 256 experts, top-8 plus a shared one, and measurably steeper than OLMoE — +39.9pp excess over null at top-26%, against OLMoE's +20.9pp at the same fraction. A hot set unambiguously exists.

Learned pinning still loses, monotonically. i9-12900K, int4-gs64 container, cap 32, 48 tokens, .coli_usage history built on prompt A:

pins	same prompt as history	held-out prompt
HOT=0 (pure LRU)	65.0% hit · 1.83 tok/s	64.0% · 1.74
HOT=8	64.0% · 1.83	63.7% · 1.75
HOT=16	63.3% · 1.74	63.0% · 1.71
HOT=32 (= cap)	60.3% · 1.70	—

Endpoints repeated three times with non-overlapping ranges (HOT=0: 1.88/1.86/1.82, HOT=32: 1.65/1.66/1.71) — about 10% slower with pins. At cap 128 and 256 the difference vanishes entirely (78.2% hit either way), because the cache holds most of the routed set regardless and placement has nothing left to decide.

We tested and rejected the two obvious explanations. Not "the whole cache is pinned so LRU has no room": the dose-response is smooth from HOT=8, which pins only a quarter of it. And not "routing is flat": measured above, it is the steepest of the three models we have traces for.

What is left is a hypothesis we have not yet tested, so stated as one. cap 32 of 256 is 12.5% of the experts, and the top ~12% already catch ~65% of routes — which is the set an LRU converges on by itself. Pinning then freezes the set LRU would have found anyway and pays for it in lost adaptivity. That predicts pinning should only win where LRU cannot find the hot set: a working set churning faster than LRU adapts, or a miss expensive enough that a small hit-rate gain outweighs the lost adaptivity.

The second condition is measurable and was not met here: on this host a miss cost 2.62 ms and it was CPU, not I/O — squeezing the cgroup from 40G to 15G against a 22 GB container changed decode by -0.015 ms/miss. So this is a datapoint against pinning in the resident regime specifically, and says nothing about the disk-bound one where the learned-pin results in the hypothesis table were originally taken.

Caveats on ours, since yours is the better-sampled study: one prompt of 14 tokens generating 48, per cell; n=3 at the endpoints and n=1 at the intermediate pin counts; skew from a single 2440-row trace whose excess was still climbing at the end, i.e. not converged. Nothing here supports a general claim about PIN=auto — only that on a 256-expert model with a genuine hot set, in the resident regime, pinning lost every way we measured it.

[lornecodes]

Correction to one line in my comment above, before anyone builds on it.

I wrote that a miss on our host "was CPU, not I/O — squeezing the cgroup from 40G to 15G against a 22 GB container changed decode by -0.015 ms/miss." The conclusion is right but that argument does not support it, and I should not have offered it.

Bracketing the same runs against /proc/diskstats shows 9.49 GB of device reads per run at every memory level, identical to two decimals. The cap never produced a differential — even 40G cannot hold a 22 GB container alongside ~12 GB of anonymous memory, so all three rungs were equally uncached and the comparison had no contrast in it. A null from an experiment with no contrast says nothing about CPU versus I/O.

What actually establishes it: vectorising the integer int4 unpack on that engine produced a 2.10x end-to-end speedup (#1271), which is not possible on a storage-bound workload. And the 9.49 GB is ~1.6 s at that drive's 5.78 GB/s — under 6% of the original 28.4 s decode — and is essentially the one-time dense-weight load rather than expert-miss traffic.

Nothing in the pinning results changes: those are direct A/Bs at fixed cap and their miss counts and hit rates are unaffected by any of this. Only the parenthetical about why misses were cheap was unsupported.