The learning cache is engine-specific: .coli_usage has two incompatible writers and two engines that cannot produce it at all

[terrizoaguimor]

Summary

"A cache that learns" is one of the first things the README promises, and routing history is what feeds it (PIN=auto, AUTOPIN, REPIN, the .coli_pairs coupling prefetch, the expert atlas in the dashboard). That machinery is per-engine today, and the four engines disagree:

Engine	ROUTE_TRACE	writes .coli_usage	reads it for PIN
colibri.c (GLM-5.2)	✅	✅ sparse text triples	✅
inkling.c	❌	✅ dense binary IKU1	✅ (own format only)
olmoe.c	❌	❌	❌
kimi_k3.c (#676)	❌	❌	❌

So: routing traces exist on exactly one engine, and the usage file is written in two mutually unreadable formats under the same filename.

I hit this while instrumenting K3 to collect routing data, and rather than send a K3-only patch I would rather propose the shared facility — if that shape appeals, I am happy to implement it.

The format collision, precisely

colibri.c writes sparse text through stats_dump_q() (telemetry.h):

FILE *f=fopen(tmp,"w");                       /* text */
... fprintf(f,"%d %d %u\n", i, e, m->eusage[i][e]);   /* layer expert count, non-zero only */

inkling.c:798 writes a dense binary block with its own magic:

FILE *f = fopen(tp, "wb");                    /* binary */
uint32_t hdr[3] = { 0x31554B49u /* "IKU1" */, (uint32_t)c->n_layers, (uint32_t)E };
fwrite(hdr, 4, 3, f);
for (int i = 0; i < c->n_layers; i++) fwrite(m->eusage[i] ? ... , 4, E, f);

Both default to <snap>/.coli_usage and both honour PIN=<path>. The comment above inkling.c's writer says "same contract as glm's .coli_usage" — it is a different contract.

What that costs today:

1. Point inkling at a snapshot where GLM has run: the magic check at inkling.c:750 fails and it declines cleanly — "[pin] …: not an inkling usage file, ignoring" — so the user does get told, but pinning is off and the history they accumulated is unusable.
2. The other direction is quieter. GLM's usage_load is while(fscanf(f,"%d %d %u",...)==3): handed inkling's binary file it parses zero triples and returns 0 with no message, which is indistinguishable from "no history yet".
3. Share a model directory between the two engines and whichever ran last has overwritten the other's history, in a format the other cannot read.

Why the fix is not "share the types"

telemetry.h already exists but is bound to GLM's structures — its own header says so:

Include after Model/Cfg/QT/ESlot/shards and st.h are defined; requires qt_bytes(), now_s(), rss_gb(), edisk_s(), and the g_cuda_* globals.

kimi_k3.c cannot include it: its Model/Cfg are different types with different fields, which is exactly why #676 is a separate engine. And #667 proposes unifying those types while #676 argues the architectures are incompatible — this proposal deliberately takes no side there. Routing telemetry does not need shared types. It needs a facility that speaks integers.

Proposal: c/route_trace.h

Dependency-free, no Model/Cfg, no st.h. Every engine already knows its layer index, the ids it selected and their gates — that is the entire input.

/* Engine-agnostic routing telemetry. Include anywhere; owns its own counters. */
static void rt_init(int n_layers, int n_experts);   /* reads ROUTE_TRACE= and the usage path */
static void rt_route(int layer, int row,            /* one call per routing decision:      */
                     const int *ids, const float *gates, int k);   /* traces AND counts    */
static void rt_save(void);                          /* atomic write of the usage file       */
static int  rt_load(const char *path);              /* read history back, for PIN/AUTOPIN   */
static const uint32_t *rt_counts(int layer);        /* what PIN ranking consumes            */

Per-engine cost after that: rt_init at startup, one rt_route in the router loop, rt_save where the engine already persists, rt_load where PIN is handled. For kimi_k3.c that is four call sites; olmoe.c likewise.

Format contract

The wire format should stay exactly what colibri.c emits today, because the ecosystem is already built on it — PIN=auto, tools/route_pairs.py, the dashboard's expert atlas, and any user's saved histories:

• trace line: <call> <row> <layer> <id>:<gate%.4f> ... (colibri.c:3028-3031)
• usage line: <layer> <expert> <count>, sparse, non-zero only

inkling.c's binary format is more compact for a dense history, so the sensible migration is: the shared writer emits text, and inkling.c's reader keeps sniffing the IKU1 magic for one release so existing histories still load. Happy to do it the other way round if you prefer the binary format as canonical — the point is one format, not which one.

Validation

The refactor of colibri.c is the only risky part, and it is provable rather than arguable:

1. ROUTE_TRACE=/tmp/a on the current build, ROUTE_TRACE=/tmp/b after the refactor, same prompt and seed → cmp /tmp/a /tmp/b must be silent.
2. Same for .coli_usage after an identical run.
3. Oracle: SNAP=./glm_tiny TF=1 ./colibri 64 16 16 → 32/32, and greedy → 20/20.
4. make check, clean build, 0 warnings, CUDA=1 and portable.

If (1) and (2) are byte-identical then PIN=auto, REPIN, .coli_pairs and the dashboard cannot tell the refactor happened.

Suggested phasing

• A. route_trace.h + colibri.c refactored onto it, with the byte-identical evidence above. No new behaviour, no new env vars. This is the PR that needs your review.
• B. kimi_k3.c — proves the facility is actually engine-agnostic rather than just claimed to be. Best landed after Kimi K3 engine: native-MXFP4 expert streaming, KDA + gated NoPE MLA + AttnRes + LatentMoE (#658) #676, or offered to @steve-m for inclusion.
• C. olmoe.c and inkling.c, plus the IKU1 transition read.

Each phase stands alone; if only A lands, the collision is documented and nothing is worse.

What this is not

• Not an architecture change, and not a vote on feat(core): model-architecture seam, chat templates, text antiprompt #667 vs Kimi K3 engine: native-MXFP4 expert streaming, KDA + gated NoPE MLA + AttnRes + LatentMoE (#658) #676 — it works with either, or with neither.
• Not new surface: it removes a duplicated writer and gives an existing feature more coverage. Same spirit as the concern raised in [Feature]: Project Complexity Reduction #660.
• Not a new dependency: pure C, no headers beyond stdio/stdlib/stdint.

Offer

I have K3 routing telemetry working already (that is how I found this), and an H100 box with GLM-5.2 resident, the oracle fixture built, and tooling that consumes both formats. If the shape above looks right I can send phase A with the byte-identical diff evidence, and phase B behind it.

[JustVugg]

The collision is real and your table is accurate. Two writers, one filename, mutually unreadable:

• colibri.c writes sparse text triples through stats_dump_q()
• inkling.c writes a dense binary IKU1 header
• olmoe.c and kimi_k3.c produce nothing at all, so PIN=auto has nothing to read

And ROUTE_TRACE exists on exactly one engine, which makes routing data collection a GLM-only capability. That's a gap between what the README promises ("a cache that learns") and what three of the four engines can do.

Yes to your offer — please implement the shared facility. You've earned the benefit of the doubt on design taste; I'd rather have your version than a K3-only patch. Two constraints and one preference:

1. Read the existing files. People have .coli_usage histories accumulated over weeks of real use, and that ranking is the whole value of PIN=auto. The new reader must accept both current formats (sniff the IKU1 magic vs text) even if it only ever writes one. Silently starting from zero would be a worse regression than the collision.
2. Keep it engine-agnostic in the header, not in each engine. The four engines share st.h, quant.h, tok.h; usage/history belongs in the same place, so a fifth engine gets it for free instead of choosing a third format.
3. Preference, not a requirement: if the format carries (n_layers, n_experts) and the model identity, a mismatched history can be refused by name rather than misread — the same refuse-rather-than-guess line we hold for weight containers (formats: self-describing container stamp (TRUST-VERIFY-REFUSE) + FORMATS.md registry (#524) #529).

If it helps, inkling.c's writer is the one I touched most recently (#701) and I'm happy to answer questions about its layout. Also worth knowing: Inkling and Kimi both stream far more experts per token than GLM (topk 6 and 16 vs 8), so their histories are denser — whatever encoding you pick will get exercised harder there than on GLM.

Thanks for raising it as a shared facility instead of patching around it. That's the second time this week you've turned something you hit into something the whole engine benefits from.

[terrizoaguimor]

All three accepted — and (1) and (3) pull against each other in a way that turns out to have a clean answer, so let me put the design down before writing code.

The tension

My original plan was byte-identical output, so nothing downstream could tell the refactor happened. Your (3) wants the file to carry (n_layers, n_experts) and model identity so a mismatch is refused by name. Those cannot both be literally true — an identity header is new bytes.

A leading comment line does not work either: both existing readers are
while(fscanf(f,"%d %d %u",...)==3), so a #-prefixed line makes the first fscanf return 0 and the loop exits having read nothing. That would be precisely the silent-zero regression you warned about in (1).

The answer: negative-layer sentinels

Both current readers discard out-of-range rows and keep going:

/* telemetry.h usage_load */   while(fscanf(f,"%d %d %u",&l,&e,&cnt)==3)
                                   if(l>=0 && l<=c->n_layers && ...)
/* colibri.c:6451 pin_load */  while(n<cap && fscanf(f,"%d %d %u",&l,&e,&cnt)==3){
                                   int ok = l>=0 && e>=0 && ...;

So a header encoded as triples with a negative layer is parsed and dropped by today's readers, and the loop continues into the real data:

-1 <n_layers> <n_experts>        # dimensions
-2 <arch_id> <format_version>    # identity
3 0 17                           # ... the same triples as today, unchanged
3 1 9

• Today's usage_load / pin_load: both sentinels fail l>=0, get skipped, real triples load exactly as now. In pin_load they also do not consume the n<cap budget, since ok is false.
• The new reader recognises the sentinels, checks dims and arch, and refuses by name on mismatch.
• inkling.c's reader still declines cleanly on the magic check until phase C teaches it text.

Net effect: your (1) and (3) both hold, and the data section stays byte-for-byte what colibri.c writes today.

For naming rather than numbering the refusal, the shared header can own the mapping — a mismatch then reads "this history was written by inkling; this engine is glm_moe_dsa — refusing (pass PIN=<path> to override)" rather than printing two hashes at the user. Four entries today, one line per future engine.

Revised validation, which is now stronger than cmp

Byte-identity of the whole file is out, so instead:

1. Strip the sentinel lines and cmp the remainder against a pre-refactor dump — the data must be identical.
2. Run the unmodified upstream binary against a new-format file and diff the resulting [PIN] placement line and pin ranking against the same binary reading an old-format file. This tests your (1) directly rather than by argument: the old reader must not notice.
3. Same for ROUTE_TRACE (no header there — it is a stream, not a history), where full cmp still applies.
4. Oracle 32/32 TF + 20/20 greedy, make check, 0 warnings on CUDA=1 and portable.

On (2), and on density

Agreed on placement: the facility goes next to st.h/quant.h/tok.h as c/route_trace.h, owning its own counters so no engine has to define storage or a format.

Your density note is the right thing to check before committing to text, so: K3 at 92 MoE layers x 896 experts is 82,432 possible entries, and the profiles I collected while instrumenting it settle around 65k non-zero — roughly 1 MB of text per history, written once per turn with an atomic rename. That is fine, and it stays human-inspectable and sort-able, which is most of why the text format has been useful. If a real history ever gets pathological, the sentinel header is where a -3 <encoding> line could later select a packed body without breaking any reader that skips negative layers.

Scope of phase A

c/route_trace.h + colibri.c moved onto it + the sentinel header + a reader that accepts legacy text, IKU1, and the new format. No new env vars, no behaviour change, no engine other than GLM touched. Phase B (kimi_k3.c) follows once #676 lands, and I will offer it to @steve-m rather than racing his branch.

Will ping here when phase A is up with the evidence from (1) and (2). And thanks for the inkling.c offer — I may take you up on it for the IKU1 dimension checks, since that reader validates hdr[1]/hdr[2] against the live config and I want the new reader to fail the same way rather than more loosely.

[terrizoaguimor]

Phase A is written and tested. Before opening the PR I want to put the evidence here, along
with four places where I had to correct myself — one of them a bug I would have shipped if I
had trusted a Linux-only run, and one where my own fix was worse than the problem.

What it is

c/route_trace.h — the history format and the trace stream, with no Model/Cfg
dependency — plus colibri.c and telemetry.h moved onto it.

c/route_trace.h              new  307  the facility
c/tests/test_route_trace.c   new  272  14 cases / 51 assertions, incl. the old-reader contract
docs/routing-telemetry.md    new  145  the format, and why the fields sit where they do
c/colibri.c               +42 -31  uses it
c/telemetry.h              +4 -16  two functions become wrappers
c/Makefile                +21 -18  deps + the new test binary
docs/tuning.md             +5  -0  links the format doc from the learning-cache section

It removes a duplicated writer and pin_load's duplicated parser; the modified files come out
at +72/-65, so roughly a wash on line count, with the duplication replaced by one tested
implementation. No new env vars, nothing printed on a healthy run.

On behaviour I want to be exact rather than reassuring, because "no behaviour change" is the
claim a reviewer has to take on trust, and mine was untrue in three places when I first
believed it was done. They are in their own section below. As it stands now: identical for
every history this codebase writes, and identical for the malformed and foreign ones I could
construct.

The header records, and the two rules they are forced by

-1 <n_layers>       <n_experts>
-2 <format_version> <engine_id>
<layer> <expert> <count>

Negative layer so both existing readers (usage_load, pin_load — each a
while(fscanf(f,"%d %d %u",...)==3) guarded by l>=0) parse and discard them and continue
into the data. Two things that are easy to get wrong, and I got the second one wrong first:

• No field may be non-numeric. A string makes fscanf return < 3, ending the loop and
  silently dropping every record after it — the exact silent-zero regression you warned
  about. That is why the engine is a hash, with the names kept in the header so a mismatch
  is still reported in words.
• The hash must be the third field. Old readers parse the second with %d, and
  glm_moe_dsa hashes to 3815245270 — above INT_MAX, so undefined behaviour there.
  I had it in the second field initially. On glibc it appears to work, which is worse than
  failing: no test would have caught it.

tests/test_route_trace.c asserts the contract against a literal copy of the old reader
loop, so either rule breaking fails CI instead of corrupting a history.

Your constraint (1), tested rather than argued

Check	Result
ROUTE_TRACE dump, before vs after the refactor	byte-identical
History data section, before vs after	byte-identical
[STATS] summary line	identical
Unmodified dev binary reading a new-format file	identical [PIN] placement
New binary reading a legacy file (no header)	identical placement
Real model (GLM-5.2, 78x256, 4,637 records): same history into both builds	same 9,112 selections, same 0 VRAM + 941 RAM placement
Dashboard EMAP line, both builds	byte-identical (76x256, 38,924 bytes)

The two row counts are both right: the history header records n_layers (78, every layer),
while EMAP emits only the layers that actually route plus the MTP row (76). Different
questions, not a contradiction.

Every line of that table was re-measured after the four fixes rather than carried over from
before them. It had to be: finding 3 changes which counter rows exist, and emap_emit
decides whether to emit the MTP row by testing whether that row exists — so the earlier
numbers were describing code that no longer exists.

An empty history stays a zero-byte file — PIN=auto decides usability by file size and
falls back to stats.txt, so a header in an empty history would have quietly cost that
fallback. Found by re-reading, not by testing.

Your constraint (3)

[USAGE] …/.coli_usage: written by kimi_k3, this engine is glm_moe_dsa
        — refusing (pass PIN=<path> to use it anyway)

Wrong dimensions, another engine, and inkling's IKU1 are each refused by name on every
path the engine picks for itself — the .coli_usage it finds next to the weights, PIN=auto,
and the AUTOPIN seed. A path the user typed is honoured instead, which is what the message
above promises and the only way to try one engine's priors on another deliberately.

IKU1 is tested in the accepting direction too (as inkling), because otherwise the binary
decoder is only ever exercised on its refusal and a broken decode would read as a pass.

Four things I had wrong

Worth reading before the code, because three of them are the kind that pass every test and
show up as a support thread six months later.

1. The Windows rename(), found by cross-compiling. I claimed the header depends only on
stdio/stdlib/string/stdint. I built the test with MinGW-w64 and ran it under wine:
11 failures. The CRT rename() fails with EEXIST when the destination exists, so the
history silently stops updating after its first write. You already fixed this once —
compat.h's shim exists because of stats_dump_q:

"dopo il primo write il file esiste gia', e CRT rename fallisce silenziosamente"

My "pure libc" header skipped the shim and reintroduced it. It now includes compat.h, with
the reason above the include so nobody removes it for tidiness. The accurate claim is
therefore libc plus compat.h — a shared platform header, self-contained, no engine
types. Still no Model/Cfg, but not what I first said.

2. The total counted records the reader had rejected. Both replaced readers added to
their total inside the same if that admitted the record, and their admission tests are
not the same — usage_load bounds the layer, pin_load bounds it implicitly through its
sparsity test. My rt_read totalled everything it parsed. A legacy headerless history from a
larger model is exactly the file that breaks this, and it carries no dimension record to be
refused by, so [USAGE] expert history: N selections came out inflated. The record callback
now returns whether it accepted, and only accepted records are totalled.

3. Dense layers absorbed counts they used to drop. The old code expressed "this layer has
no experts to attribute a count to" by leaving eusage[i] NULL on dense layers — the && m->eusage[l] in usage_load was doing double duty as a bounds check and the admission
rule. rt_init allocates every row, so that filter disappeared and a record naming a dense
layer was absorbed, totalled, and written back out. Added rt_drop_row, which the engine
calls for its non-routing layers once sparsity is settled, restoring the exact shape.

4. A refusal message that advised a workaround that could not work. Every identity
refusal says "pass PIN=<path> to use it anyway" — but pin_load reads through the same
rt_read, so it hit the same check and refused again. Following my own message got you
nowhere, and the old pin_load checked nothing, so it was a regression too.

Fixing it took two tries, and the first was wrong in a way worth describing because my own
QA caught it. I made trusted reads skip every check, and three refusal tests went red.
Reading them told me the distinction I had missed: trust follows how the path was reached,
not what the file contains. PIN=<file> was typed by a person. PIN=auto and the
AUTOPIN history are paths the engine found by itself — nobody vouched for those, and my
first version would have let a foreign .coli_usage drive placement with no refusal at all,
which is the exact hole constraint (3) exists to close. pin_load now takes that as an
argument, because one function serves both kinds of path and the file cannot tell you which.

And trust relaxes only the identity check. A format version, and the dimensions of an
IKU1 file where a record's layer comes from its position, are parse geometry: getting them
wrong is misreading rather than misattributing, and neither refusal ever promised a way
past. So a named file is honoured; a file this build cannot parse is still refused.

Those three red tests were, I should say, testing the old wrong behaviour — they asserted
PIN= refuses. I rewrote them to assert what the guarantee actually is: the automatic paths
refuse and name the writer, and the named path is honoured. Both directions are now covered,
where before only one was.

Where these came from is the part I would want to know as a reviewer. The Linux suite was
green for all four of them. 1 needed a second platform; 2, 3 and 4 came from re-reading the
diff against the code it replaced. The suite did catch something real — the bad first fix for
4 — which is the useful pattern: tests are good at telling you a change broke a rule you
already wrote down, and no help at all when the rule was never written down. That is why the
contract test asserts against a literal copy of the old reader loop rather than against my own
expectations of it.

Phase B, proven as far as it can be without the model

I applied the call sites to the pristine kimi_k3.c from #676. It compiles with no new
diagnostics — the one warning in that file (-Walloc-size-larger-than= on
calloc(c->n_layers, sizeof(LCache))) is present in the pristine file at the same site, which
I checked by building both with identical flags rather than assuming. The whole diff:

+#include "route_trace.h"
+rt_init("kimi_k3", m.c.n_layers, m.c.n_experts);
+for(int i=0;i<m.c.n_layers;i++) if(!m.L[i].sparse) rt_drop_row(i);
+rt_drop_row(m.c.n_layers);           /* K3 has no MTP row */
+rt_load(up)                          /* when COLI_USAGE is set */
+rt_route(li, t, idx, wsel, Kt);      /* traces and counts, one call */
+rt_save(up, 0);

Five call sites, on a Model and Cfg that share nothing with GLM's. The drops are there
because K3 has dense layers too (l->sparse = (i >= c->first_dense)) and no MTP row — which
is the more interesting result of the exercise: the row-shape problem from finding 3 was
not a GLM quirk, it recurs, and adopting the facility surfaced it in the second engine
immediately rather than after someone shipped a corrupted history.

I have not run it. The K3 container was on a box I have since destroyed, and re-fetching
1.5 TB to strengthen a claim that is already at compile level seemed like the wrong trade to
make unasked — say the word and I will. Otherwise this belongs in its own PR after #676 lands,
and I would rather offer the patch to @steve-m against his branch than race him to it.

Not covered

inkling still reads only IKU1 — it declines a new-format file cleanly, which is correct
until phase C teaches its reader the text layout. olmoe.c is untouched.

What was run

Refactor suite (glm_tiny, incl. make check and both builds warning-free)	19 / 19
End-to-end on the real GLM-5.2 (~400 GB)	6 / 6
tests/test_route_trace on Linux	14 cases, 51 assertions, pass
Same test cross-compiled with MinGW-w64, under wine	pass
kimi_k3.c from #676 with the call sites	compiles, no new diagnostics

Happy to open the PR against dev whenever you want it, or to change the shape first if any
of the above reads wrong to you. The one thing I would ask you to look at even if you skip
the rest is the trust rule in 4 — whether PIN=<file> overriding an identity mismatch is
the behaviour you want, since that is a product decision I made and then had to correct once
already, not something the old code can tell us.

[JustVugg]

Both of your comments deserved an answer sooner than this — the design one and the phase-A evidence have been sitting while I was elsewhere in the tracker. Answering all three open questions.

The trust rule (finding 4) — yes, keep it, with one addition

You are right that this is a product decision the old code cannot settle. Here is the reasoning I want on the record, because it is a different rule from the one we hold for weight containers, and the difference has to be justified rather than assumed.

#529 refuses a mismatched weight container absolutely, with no override. A wrong container changes what the model computes — different tokens, and nothing tells you. There is no legitimate reason to want that, so there is no escape hatch.

A placement history cannot do that. Pins and residency decide which bytes are in RAM, never what is computed. Feed an engine a foreign history and the worst case is that it pins useless experts and runs slower: self-inflicted, bounded, and visible in the hit rate. Wrong output and wrong speed do not deserve the same rule.

That asymmetry is what makes your split correct — automatic paths refuse because nobody vouched for them, a typed path is honoured because someone did. It is also the only way to keep questions like "do GLM's priors transfer to another engine at all?" askable, which is a real experiment (see #708) and impossible under a hard refusal.

The addition I would want: when a typed path overrides an identity mismatch, say so, loudly and once, rather than accepting it silently.

[USAGE] …/x.coli_usage: written by kimi_k3, this engine is glm_moe_dsa
        — honouring it because you named it explicitly
          (placement only; expert ids do not transfer between engines)

Two reasons. A typo'd path and a deliberate cross-engine experiment are indistinguishable at the call site but not to the user, and that line is the only thing separating them. And when someone posts a benchmark to this tracker it will be in their paste — a run with foreign pins that looks mysteriously slow should carry its own explanation.

The second half you already have right: trust relaxes identity only. Format version and geometry are parse correctness rather than attribution, and no refusal ever promised a way past them. Keep that exactly as it is.

Phase A — open the PR against dev

Nothing in the shape reads wrong. The negative-layer sentinel is a better answer than either of the things I asked for, because it delivers (1) and (3) simultaneously instead of trading them, and the reserved -3 <encoding> line means the format is not a dead end if a history ever does get pathological.

The four corrections are why I am not asking for more evidence:

• Findings 2 and 3 are exactly the class of bug that byte-identity is supposed to exclude and does not: the file is identical and the numbers reported about it are not. Finding them by re-reading the diff against the code it replaced, after the suite was green, is the part I would have missed.
• Finding 3 recurring in kimi_k3.c is the strongest evidence in your whole comment that the facility is genuinely engine-agnostic. A shared header that only ever fits the engine it was extracted from is the normal failure mode of this refactor; yours surfaced a row-shape problem in the second engine before it could ship a corrupted history.
• Finding 1 is a debt of ours, not yours. compat.h exists precisely because CRT rename() fails on an existing destination, and it cost us a silent stall once already. That a fresh header written by someone who had clearly read the codebase still reintroduced it means the shim is not discoverable enough. That is a docs bug on our side and I will take it.
• The glm_moe_dsa → 3815245270 > INT_MAX catch is the one I would have shipped. Working on glibc is worse than failing, and no test written against expectations rather than against the old loop would have caught it.

Asserting the contract against a literal copy of the old reader loop is the right instinct, and I would like that pattern to spread — it is the only kind of test that survives someone rewriting the thing it guards.

Phase B — #676 landed, so there is nothing to race

@steve-m's Kimi K3 engine merged into dev on 2026-07-29; c/kimi_k3.c is on dev now. Your five call sites can go in as their own PR against dev directly. A heads-up to @steve-m on that PR is courteous since it touches his router loop, but there is no branch to coordinate against any more.

And no — do not re-fetch 1.5 TB. Compile-level proof plus the drop-row shape result is the right trade for a facility that cannot change model output, and spending that much transfer to strengthen a claim already established at compile level is not a cost I am willing to have you absorb. If it turns out to matter, Kimi is reachable here and I will run it.

One coordination note

You are about to own this file, so: #708 proposes an expert-transition-history placement policy — order-1 Markov over (layer, expert) transitions — built on exactly the history you are formalising. @matthewworner will likely need a per-transition record rather than the per-expert counts .coli_usage holds today, and your reserved -3 line is where that belongs. I have pointed him here. Worth the two of you agreeing the record shape before either writes it, so the format grows once instead of twice.

Phase A whenever you are ready. And while you are in the area, #696 is still waiting on a rebase.

[JustVugg]

Confirming plainly, so there is nothing left to read between the lines: a single shared .coli_usage across all engines is exactly what we want, and phase A is approved. Open the PR against dev.

One format, one reader, one writer, living next to st.h / quant.h / tok.h so the next engine inherits it instead of inventing a third layout. That is the right shape for this project, and you got there yourself from a problem you hit — which is the best way for it to happen.

Phase B (kimi_k3.c) as its own PR behind it, phase C (olmoe.c, inkling.c + the IKU1 transition read) whenever you have the appetite. No need to ask again for either.

And more generally, for anyone reading this thread: you are welcome to try. If you see something that would make colibri better — a gap, a bug, a facility that should exist — propose it and send the PR. You do not need permission first and you do not need to be sure it will work. This issue is a good example of why: it started as someone instrumenting a different engine for their own reasons, and it is going to leave the whole project with a capability three of the four engines did not have. Negative results are welcome on the same terms.

[terrizoaguimor]

Thank you — and one correction first, because it is the kind that costs someone else work if
it sits.

There is no reserved -3 <encoding> line. I never specified one.

You referred to it twice, and pointed @matthewworner here on the strength of it. What the
format actually has is the general mechanism, which is weaker than a reserved record but
covers what you want from it:

if(l < 0) continue;          /* room for future header records */

Every reader — including the two that predate this header and are still out in older binaries
— advances past any record whose first field is negative, because each is a
while(fscanf(f,"%d %d %u",...)==3) guarded by l>=0. So a new record type can be added at
any time and older builds skip it rather than break, provided it keeps to the two rules:
three fields, all numeric.

What is not defined is any specific record beyond -1 and -2. No number is claimed, no
field meanings are fixed. I would rather say that plainly than let @matthewworner design
against a spec that does not exist and discover it at implementation time. I have documented
the mechanism and its one constraint in routing-telemetry.md, and left the record itself
open.

@matthewworner — on the substance of #708, the question worth settling before either of us
writes anything: a per-transition count over (layer, expert) → (layer+1, expert') is
inherently four-dimensional, and this format's records are three numeric fields. Two ways out:

1. Pack the pair. -3 <src_layer*n_experts+src_expert> <dst_expert> as the key with the
   count in a field, which fits three fields but only if the destination layer is always
   src_layer+1 — true for order-1 adjacency, false the moment you want skip-layer pairs.
2. Two records per transition. Split it across two three-field lines rather than widening
   one.

I had written a third option here — widen the record and bump RT_FORMAT_VERSION — and then
tested it before posting, because I was about to hand you a constraint you would build on.
It is not viable, and the way it fails is the exact failure this format exists to prevent.
A four-field record does not merely get skipped by an old reader. It desynchronizes it and
fabricates an admitted record out of the leftover field plus the start of the next line:

file:    3 2 5        -3 7 12 99        4 1 8
old reader sees:
  l=3  e=2 c=5    admitted   (correct)
  l=-3 e=7 c=12   skipped    (correct — the guard works)
  l=99 e=4 c=1    ADMITTED   <- invented: the leftover 99, then two fields of the NEXT line

The real 4 1 8 is eaten, and a count lands on layer 99. Silently, in every binary already
shipped. So the rule is exactly three numeric fields, never more — not "at least three",
which is what I would have told you from reading the loop rather than running it. I have put
this in routing-telemetry.md with the same demonstration, since it is the one property of
this format that a well-meaning change can break invisibly.

That leaves 1 and 2. I lean towards 1 if order-1 adjacency is the whole scope: it needs no
version bump, no ordering rule, and no second record type. Your call — you know what the policy
needs. Whichever we pick, the per-expert counts stay exactly as they are: a transition history
is additional data, not a replacement, and PIN= must keep working on a history that has none.

The override notice — implemented as you specified

[USAGE] /tmp/foreign.coli_usage: written by kimi_k3, this engine is glm_moe_dsa
        — honouring it because you named it explicitly
          (placement only; expert ids do not transfer between engines)

Once per process, not once per read, so a REPIN cycle that re-reads the file does not turn it
into noise — and the guard is on the message only, asserted so that a repeated trusted read
still returns the same records. The end-to-end check requires the line to appear exactly
once and to carry the scope clause, because a notice that fires per record would be
self-defeating for the paste-into-the-tracker case you described.

Adding the notice broke two of my own checks, in a way worth passing on. The notice contains
written by kimi_k3 — the same substring as the refusal it replaces — so a check written as
"if the output mentions the other engine, it was refused" now reports a successful override as
a failure. I spotted it in one check, fixed it to key on refusing versus honouring, and
then the adjacent check failed on the identical mistake four lines below. The lesson is not
that I was careless twice; it is that matching on the engine name was never testing what it
claimed to test, and it took the message changing to reveal that. It would have gone on
reading green indefinitely.

Your framing of the asymmetry is better than the one I had. I argued from what the message
promises; you argued from what the two kinds of mismatch can actually do — a wrong weight
container changes what is computed and nothing tells you, a wrong history changes which bytes
are resident and the hit rate tells you. That is the real reason #529 gets no override and this
does, and it is now the comment above the function rather than my weaker version.

Phase A

#716, against dev. Two notes on getting there.

My base was 853 commits stale — I had been working from before the Vulkan backend and the
olmoe chat mode, with colibri.c +982/−147 behind. I rebased rather than submit against that:
colibri.c, telemetry.h and tuning.md applied cleanly to 292ed4c, and the Makefile
needed two resolutions, both mechanical — your TEST_BINS list and your colibri target now
carrying $(VK_OBJ)/$(VK_SPV), in which case the answer is your line plus my one dependency,
not a choice between them. The full suite has been re-run on the rebased tree rather than
carried over, since 982 changed lines of colibri.c is not something to assume through.

The rebase produced a piece of evidence I could not have manufactured. Your Vulkan
expert-selection path is a third consumer of m->eusage[i]-as-a-NULL-test:

for(int i=0;i<NL;i++) if(m->eusage[i]) for(int e=0;e<E;e++) if(m->eusage[i][e]) nz++;

It was written after my branch, by someone who had never seen this header, and it depends on
exactly the row-shape property that finding 3 was about. Had I shipped the version that
allocates every row, this is code that would have started walking dense layers. That the
convention re-appears independently in a new backend is a better argument for rt_drop_row
than anything I wrote to justify it.

One further doc fix beyond the notice: it still claimed the header depends only on libc, which
I had corrected in prose here but not in the file that ships. It now states libc plus
compat.h with the rename() reason inline — your point about the shim not being
discoverable enough is exactly right, and a doc repeating my original wrong claim would make it
worse.

On compat.h being a debt of yours rather than mine: I would not have found it either without
cross-compiling, and the shim's comment does say why it exists. What it does not do is sit
anywhere a new header author would look. If a line in CONTRIBUTING.md under "writing a new
header" would help, I am happy to send it with phase B.

Phase B and #696

#676 on dev — good, the five call sites go in as their own PR and I will tag @steve-m on it
since it touches his router loop. Noted on not re-fetching; I will take you up on running it
there if the transition work in #708 ends up wanting a second engine's history to test against.

#696 is mine and it is conflicting — that is on me, it predates all of this. I will rebase
it on dev before phase B so I am not stacking new work on top of a stale PR of my own.

[terrizoaguimor]

One small process note, offered in the same spirit as the rest of this thread.

The -3 <encoding> line is the case in point, and it cost nothing — but it cost nothing
because I happened to check before accepting the credit, which is luck about who read it
rather than anything structural. Had I taken it at face value, @matthewworner would have
designed against an interface that does not exist, found out at implementation time, and the
two of us would have spent a day reconciling a disagreement neither of us introduced.

I want to be even-handed, because I did the same thing repeatedly in this issue and it was
mostly my own QA that caught me:

• I claimed the header depended only on libc. It depends on compat.h — Windows told me that,
  I did not work it out.
• I claimed "no behaviour change" three separate times before it was actually true.
• I was about to tell @matthewworner that a wider record "gets skipped by old readers." I
  tested it before posting and it does not: it desynchronises the reader and fabricates an
  admitted record. That one would have cost him real work, and I nearly sent it.

So the ask is not "you were wrong about -3." It is that on this format specifically, it is
worth confirming before pointing a third party at an interface — in either direction. A
history file is read by binaries already in the field, so a wrong assumption about it does not
fail loudly. It lands as a count on layer 99 and nothing reports it. That property is precisely
what makes confirming cheap and assuming expensive.

Concretely for #708: before @matthewworner or I write a line of the transition record, one of
us states it here — record number, fields, meanings, ordering — and the other says yes or
adjusts it. Ten minutes, and the format grows once instead of twice.

None of which takes away from the review itself. Your asymmetry argument — a wrong container
changes what is computed, a wrong history changes only what is resident and the hit rate shows
it — is better than the reasoning I had, and it is in the code now instead of mine. And
"you are welcome to try" is the reason I sent a second comment rather than stopping at the
first. Thank you for that.

[terrizoaguimor]

@matthewworner @JustVugg — the record shape for #708, so it can be settled before either of us
writes code.

First: the two options I gave you earlier were both worse than they needed to be. "Pack the
pair and accept order-1 adjacency" and "spend two records per transition" were me reasoning from
the constraint rather than designing against it. Here is a third that costs neither.

The constraints, restated so the design is checkable

1. %d %d %u — exactly three fields, first two signed, third unsigned 32-bit. A fourth field
   desynchronises every deployed reader and makes it invent an admitted record.
2. The first field must be negative. A transition record with a positive first field would be
   read by existing binaries as a per-expert count and silently added to their counters. This is
   not optional.
3. Counts must not need packing, because a packed count has a ceiling and the ceiling is invisible
   until someone crosses it.

Proposal: a cursor record plus entry records

-3 <src_layer * n_experts + src_expert>  <dst_layer>     open a group
-4 <dst_expert>                          <count>         one successor, repeatable

-3 names the source and the destination layer; each following -4 names one destination
expert and its count. A group ends at the next -3, at any other record, or at EOF.

For order-1 adjacency dst_layer is simply src_layer + 1, but naming it explicitly costs
nothing and means skip-layer coupling is expressible later without a new record type or a version
bump. Worth having, since COUPLE= already prefetches across a configurable layer distance.

What this buys over both of my earlier suggestions:

• No packing. Counts are a full uint32, destinations are a full field. Nothing has a hidden
  ceiling.
• Amortised. One cursor per source, not one record per transition — so the overhead is
  1 + k records per source rather than 2k.
• Truncation-safe if you write successors hottest-first: a partial read keeps the most useful
  edges rather than a random subset.
• Invisible to everything already deployed. Both records have a negative first field, so
  usage_load, pin_load and rt_read as it stands today all skip them.

A reader that understands them is small:

static int64_t g_src = -1, g_dst_layer = -1;
...
if(l == -3){ g_src = ver; g_dst_layer = (int)v3; continue; }
if(l == -4){ if(g_src >= 0) trans_add(g_src, g_dst_layer, ver, v3); continue; }

Two things to be aware of before you commit to it

Statefulness. -4 means nothing without a preceding -3. Any reader that handles -4 must
handle -3, and should reject a -4 that arrives without one rather than attributing it to
whatever came before. Old readers are unaffected because they skip both, but it is a real property
of the format and I would rather name it than have it discovered.

Size. Top-8 successors for every (layer, expert) on GLM-5.2 is 78 × 256 × 9 ≈ 180k records,
roughly 2.7 MB — against a current history of about 4,600 records. That is fine on disk but it is a
40× file, and stats_dump_q rewrites the whole file every serve turn. @JustVugg was content to
live with the full rewrite at the current size (limit 2 in #717); at 2.7 MB per turn that decision
may deserve revisiting, or the writer should store only sources above a heat threshold. Worth
deciding deliberately rather than discovering it in a profile.

What I would suggest doing first

@JustVugg's advice in #708 is right and I would follow it: start with the pin/residency arm,
which needs only the three-field per-expert counts that already exist and is therefore not blocked
on any of this. If that arm shows the policy is worth having, the transition records are a small
addition on top and we will have learned what they actually need to carry.

I have deliberately not implemented any of this. If the shape looks right to both of you I will add
the two record types to route_trace.h with the writer, the reader and the contract test, so
@matthewworner has one thing less to build — or if you would rather own it directly, say so and I
will stay out of the way and just keep the format honest.

One more: whatever we agree, the three-field rule needs a test that fails loudly if someone widens
a record later. tests/test_route_trace.c asserts it at the writer today; the transition writer
should be covered by the same assertion rather than a new one.

[terrizoaguimor]

Correcting a number in my comment above before anyone plans against it.

I wrote that transition records would be "roughly 2.7 MB — against a current history of about
4,600 records… a 40× file." Both figures are wrong, and the error inflates my own argument.

I took 4,600 from a history written by a single short run, not from a steady state. Measured on
the actual accumulated .coli_usage for GLM-5.2:

	I said	measured
current history	~4,600 records	19,341 records / 186 KB
transition records	~180k	170,496 (74 source layers × 256 × 9)
ratio	40×	8.8× by record, 10.7× by size (~2.0 MB)

The history is also close to its own ceiling already — 76 rows × 256 = 19,456 possible, and it is
at 19,341 — so it does not grow much further with use. My "78 layers" was loose too: GLM-5.2
routes on layers 3–77, which is 75 sparse layers and therefore 74 sources with a successor.

The conclusion is the same in direction and much weaker in force: transitions are roughly an order
of magnitude larger than the current history, not forty times, and 2 MB rewritten per serve turn
is a smaller concern than I made it sound. @JustVugg — the suggestion that you revisit the
full-rewrite decision was resting on the inflated number, so treat it as an observation rather
than a recommendation until someone sees it in a profile.

Everything else in the proposal holds, and I checked the part that actually matters rather than
reasoning about it: a file mixing -3/-4 records with ordinary counts, run through a literal
copy of the reader in telemetry.h and pin_load on current dev, admits exactly the three real
records and totals them correctly. The extension is invisible to what is already deployed.

[terrizoaguimor]

@JustVugg — the windowed-aggregation design, before I write it, since #708 wants to extend the
same format and it should grow once.

What it has to solve

docs/tuning.md says it plainly: the persistent history is not decayed. A machine that ran a
coding workload for a month and switches to prose carries a month of stale priors, and REPIN's
session heat map compensates within a session and is then discarded.

The shape

One new record type, three numeric fields, negative first field so every deployed reader skips it:

-5 <layer * n_experts + expert>  <recent_count>

Written alongside the existing lifetime records, never instead of them. The lifetime totals stay
exactly as they are — same records, same bytes, same meaning — so nothing that reads the history
today changes behaviour, and a build that does not know -5 simply gets the history it gets now.

Recency without floats. recent is updated once per save with an integer EMA:

recent = recent - (recent >> RT_DECAY_SHIFT) + delta_since_last_save;

RT_DECAY_SHIFT = 6 gives α ≈ 0.984 per save and a half-life around 44 saves. No floating point,
no hidden scale factor, no overflow path, and %u holds it exactly. Computing the delta needs one
snapshot array in memory — 19,456 × 4 bytes on GLM-5.2, so 78 KB.

Size. Only non-zero entries are written, same as today, so the ceiling is one extra record per
routed expert: at most 2× the file. On a saturated GLM-5.2 history that is 19,341 → ~38,700 records,
around 380 KB. That is a real increase but a bounded one, and unlike the transition records in
#708 it does not multiply by a successor count.

Why not decay the counts themselves

Because it is irreversible. Once the lifetime totals are decayed you cannot ask a long-horizon
question again, and you cannot recover the answer you threw away. Two series cost one extra record
type and keep both questions askable. If the recency signal turns out to be useless you delete the
-5 records and lose nothing.

Why not a ring of epochs

I considered -5 <epoch> <n> as a cursor with -6 <key> <count> entries, mirroring the
-3/-4 shape I proposed for transitions. It buys arbitrary window queries after the fact, and it
costs K× the file plus the statefulness that comes with cursor records. Given the actual question —
"is this expert hot lately" — one decayed series answers it at a fixed cost. I would rather not
pay K× for generality nobody has asked for. Worth saying I could be wrong about that, and it is the
part of this design I would most like a second opinion on.

One thing that makes this more interesting than housekeeping

The measurement I posted in #717 says a domain-conditioned placement prior beats a single global
profile by +8.1 pp of decode hit rate at 32 slots per layer — and that using the wrong
domain's prior costs about twice what the right one gains, which is why abstention has to be the
default.

A recency window is domain conditioning without the classifier and without that risk. If what
you have been doing lately is writing prose, the recent series is the prose profile — nothing has
to identify the domain, and there is no wrong-domain case to abstain from, because the signal is
your own last N turns by construction. It will not match a good classifier on a workload that
alternates sharply between domains within a session. It should get most of the way there on the
common case, which is a machine used for one kind of work at a time.

That also makes it measurable against the same held-out traces I already have, so I can tell you
what it is worth before you decide whether PIN=auto should prefer it, blend it, or ignore it.

What I would send

route_trace.h gains the -5 writer, the reader, the decay step in rt_save, and a contract test
asserting that a history containing -5 records is byte-equivalent for an old reader to the same
history without them. No consumer changes in this PR — the records get written and read back, and
what PIN does with them is a separate decision I would rather make with numbers than with a guess.

Does the shape look right, and do you want the -5 records on by default or behind an env flag for
a release or two?