bench: report minimum + gcsample heavy benchmarks to cut CI variance#18
Merged
Conversation
Reduce the run-to-run and cross-runner variance that has been making the benchmark CI flag phantom regressions. Save minimum(results) instead of median(results). The minimum is BenchmarkTools' recommended tracking estimator: measurement noise (GC pauses, scheduler preemption, frequency scaling) only ever adds time, so the minimum is the most reproducible estimate of the true cost and the least sensitive to hardware differences between runners. Set gcsample=true on the two allocation-heavy ODE-solve benchmarks (Correlations two-time, Interaction Picture numerical mode evolution). They rebuild operators every ODE step and spend ~50% of their runtime in GC, which gives a ~4x run-to-run spread; running a GC before each sample brings that down to ~1.4x. The fast ns/us benchmarks are left untouched: their minimum is already stable and gcsample would only cost samples.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark CI keeps flagging phantom regressions on
Correlations/two-time(and the ODE interaction-picture one) because those benchmarks are GC-dominated. They allocate ~0.5 GB per call rebuilding operators every ODE step, spend ~half their runtime in garbage collection, and so swing ~4x run to run even on one machine. Comparing one noisy number against another across two different GitHub runners is all the "1.47x slower" alert ever was.So this just changes how the timing is measured: save the
minimuminstead of themedian(the minimum is the most reproducible number, since GC and scheduler noise only ever add time), and setgcsample=trueon the two heavy ODE benchmarks so a GC runs before each sample instead of leaking between them.Measured on the correlation benchmark, same machine, one session:
Reporting the minimum on top of that pins it near 140 ms instead of bouncing around 160 to 640 ms.
I left the fast ns/µs benchmarks alone (their minimum is already stable, 120 ns with or without gcsample), so a per-sample GC there would only cost samples for nothing.
tune!doesn't help at this scale either; it just picks evals=1.One heads-up: median to minimum is a one-time change, so the first run compares new minimums against the old median baseline and everything looks faster (nothing flagged), then the baseline resets and later runs are apples to apples.