Guard the perf work in CI, exactly where it can be exact

[JohnCampionJr]

Two guards for the work in #38, held to deliberately different standards — because one of these things can be measured exactly on a shared CI runner and the other cannot.

The struct layout is a unit test

BufferCell holds no managed references, and is 24 bytes. Neither is a measurement — they hold or they do not — so they run in the ordinary test job and cost nothing.

The reference one is load-bearing. GC write barriers are all-or-nothing: a single string field added back to the cell makes the collector trace the whole scrollback and makes every cell write emit a barrier, taking a 240-column fill from 70 ns to 239 ns. Nothing else in the suite would notice, and the failure message says what to do instead.

Throughput is a comparison, not a threshold

PerfCompare.yml builds the branch and the commit it forked from, runs one harness against both by assembly substitution, alternating, and compares medians.

Allocation is gated exactly. Bytes allocated for a fixed amount of work is a count: it does not care what else the machine is doing, how warm it is, or what the clock is. Across every calibration run below it came back byte-identical.

Time is gated against the noise the job observes in itself. That is not a preference — it is what the calibration showed. Comparing a build against itself on a quiet laptop:

work per corpus	apparent Δ on scroll-ascii	spread
60M chars	+27%	±30%
300M chars	+0.2%	±1%

A fixed threshold would have to sit above 30% to survive the first row, which is far too loose to catch anything real. The first row is also where scroll-ascii read 3.9 ns/char against the 1.7 it actually runs at — the work was too short to finish warming, which is the same trap that cost real time earlier in this project.

So the gate is max(5%, 3 × observed spread). A quiet machine earns a tight gate, a busy one raises its own bar instead of crying wolf, and the band between the floor and the gate is reported as worth a look rather than disappearing.

Checked against a real regression

Not assumed to work. Removing the _placeholderCell guard from Print — the 12% found by hand while merging Kitty, fixed in #39 — reproduced as:

corpus	bytes/char	ns/char	Δ time	noise	gate
scroll-ascii	0.00 → 0.00	1.62 → 1.64	+1.5%	±3%	9%
sgr-churn	2.33 → 2.33	4.28 → 4.38	+2.2%	±4%	11%
truecolor	1.72 → 1.72	4.44 → 4.64	+4.5%	±4%	11%
alt-redraw	0.26 → 0.26	5.37 → 5.97	+11.2% ⚠️	±2%	7%
unicode	7.66 → 7.66	12.38 → 13.46	+8.7%	±3%	10%
flood	0.00 → 0.00	40.21 → 39.70	-1.3%	±3%	10%

The right corpus, and the other five silent.

Two things that keep the comparison honest

The ci mode touches only Terminal, TerminalOptions and Write(string). That is what lets one build of the harness measure an older library — anything newer would fail at run time, and the job could then only ever compare a build against itself.

It reports the module version id of what it loaded. The path and the assembly version are both identical by design when the comparison works by swapping a file, so neither can answer "are these two different builds". If the MVIDs match, the report says so and fails rather than reporting a flawless result that means nothing.

Running it by hand

dotnet run --project src/XTerm.NET.Bench -c Release -- ci --out head.json
dotnet run --project src/XTerm.NET.Bench -c Release -- compare --base a.json b.json --head c.json d.json

The workflow is workflow_dispatch-able with chars and runs inputs, so a suspicious result can be re-run with more work — which narrows the noise column and tightens the gate automatically.

Notes

• It posts the table to the job summary always, and to the PR as a comment on a best-effort basis: a PR from a fork gets a read-only token and cannot comment.
• Defaults are 150M chars × 3 runs per side, which is a few minutes. Both are dials.
• It fails the job on a regression. Happy to make it advisory instead if you would rather see it run for a while first.

1157 tests pass.

🤖 Generated with Claude Code