perf: take advantage of upcoming algebraic operators in Rust 1.98

[jqnatividad]

By default, the Rust compiler is conservative with float operations to minimize rounding errors and to make sure float operation results are the same in debug and release mode.

The upcoming Rust 1.98 introduces algebraic operators that will unlock additional performance options for the compiler - this will accelerate float operations and make the stats and frequency commands faster still.

https://pythonspeed.com/articles/faster-float-math-rust/
https://doc.rust-lang.org/std/primitive.f32.html#algebraic-operators
https://releases.rs/docs/1.98.0/

[jqnatividad]

Rust 1.98 is now in pre-release testing
https://blog.rust-lang.org/inside-rust/2026/08/19/1.98.0-prerelease/

[jqnatividad]

1.98 is live!

https://blog.rust-lang.org/2026/08/20/Rust-1.98.0/

[jqnatividad]

qsv-stats findings: algebraic operators investigated, prototyped & validated on stable 1.98.0

TL;DR: Real but targeted win. Prototyped an opt-in algebraic cargo feature in qsv-stats that converts only the unsorted.rs reduction passes (gini / kurtosis / atkinson / mean sum). Measured 5–7× on the sequential paths, ~1.7× on the rayon paths on stable 1.98.0 (Apple Silicon), asm-confirmed vectorization. The crate's hottest function for qsv's workload (Welford's OnlineStats::add_f64) is structurally immune — and naive conversion there is a 2.5× regression. End-to-end qsv stats impact is estimated at low single digits.

What 1.98 actually gives us

{f32,f64}::algebraic_{add,sub,mul,div,rem} set LLVM's reassoc/nsz/arcp/contract fast-math flags — but not nnan/ninf, so qsv-stats' NaN-skipping semantics stay valid. Reassociation lets LLVM break the sequential FP-add chain in reductions into multiple accumulators + SIMD.

Measured (A/B harness, same toolchain, min-of-N, aarch64)

pass	scalar	algebraic	Δ
gini fused, n=5k (sequential)	0.0037 ms	0.0007 ms	5.3×
kurtosis w/ precalc, n=5k (sequential)	0.0035 ms	0.0005 ms	7×
gini fused, n=1M (rayon)	0.142 ms	0.084 ms	1.7×
kurtosis w/ precalc, n=1M (rayon)	0.121 ms	0.070 ms	1.7×

Asm attribution: the gini/kurtosis symbols go from 0 → 98 packed-f64 accumulator ops (fadd.2d/fmla.2d). The speedup is dominated by breaking the fadd latency chain (~8 accumulators), not vector width — so don't expect AVX-512 to multiply it much further.

The trap: Welford cannot be converted naively

OnlineStats::add_f64 is a loop-carried recurrence (mean_{i+1} = f(mean_i)), not a reduction — reassoc can't break it, and qsv calls it once per cell anyway. Worse: arcp folds delta * (1.0/n) into delta / n, moving a ~10-cycle divide into the recurrence chain → measured +153% (2.5× slower). A careful transcription that keeps the reciprocal off the chain gains only ~7% (single-ISA). online.rs is left untouched, with a warning comment so nobody "finishes the job" later.

Accuracy: better, not worse — but not bit-identical

Multi-accumulator summation ≈ pairwise summation: vs a Kahan reference at n=1M, the algebraic sum is ~13× more accurate than the sequential fold (1.67e-15 vs 2.10e-14 rel err). The real cost is reproducibility: results may differ in the last ~1-2 significant digits across toolchains/targets/lengths. Relevant to qsv: stats output is cached and has expected-output tests, so a cached-vs-fresh divergence is possible after a toolchain bump. That's why this is an opt-in feature, default off — feature-off codegen is unchanged (incl. single-rounding FMA), and qsv-stats MSRV stays 1.96.

End-to-end expectation

Per 1M-row numeric column: the median/quartiles sort costs ~11 ms vs ~2 ms for all FP reduction passes combined (~1.7 ms saveable). Add CSV parsing, str→f64, string columns, and I/O, and the honest end-to-end qsv stats estimate is low single digits %. Worth having since it's nearly free behind a feature flag — not worth an MSRV bump on its own.

Status / next steps

• qsv-stats prototype: algebraic feature, 167/167 tests pass feature on+off (debug+release, stable 1.98.0), clippy clean → PR to follow
• Retune PARALLEL_THRESHOLD when the feature is on (5–7× faster sequential kernel moves the rayon crossover up ~an order of magnitude)
• x86 verification (all numbers above are aarch64/NEON)
• End-to-end qsv stats --everything --force benchmark with features = ["algebraic"] to replace the estimate with a measurement
• Side finding, independent of 1.98: on baseline x86-64 (plain cargo install qsv), f64::mul_add compiles to a libm call in the hot loop. qsv's official builds are unaffected (-C target-cpu=native), but generic crates.io installs pay it.

🤖 Generated with Claude Code

[jqnatividad]

Follow-up: hasher shootout for Frequencies — verdict: keep foldhash

While closing out the pre-0.55.0 perf sweep, we also A/B'd hashbrown's default hasher (foldhash 0.2) against five challengers on qsv-frequency-shaped workloads: 1M rows/shape, HashMap<Vec<u8>,u64> built via entry_ref (mirroring Frequencies::add_borrowed), -C target-cpu=native, min-of-7, Apple Silicon, rustc 1.98.0.

shape (1M rows)	foldhash (current)	rapidhash	gxhash	fxhash	ahash	xxh3 (twox-hash)
low_card_5000 (category_NNNN)	5.25 ms	+6.5%	+10.1%	+8.4%	+10.7%	+255%
timestamps (19B)	56.80 ms	−1.5%	−3.0%	−2.8%	+5.7%	+105%
hi_card_short (8–16B hex)	37.28 ms	−2.8%	+5.4%	−1.8%	+3.5%	+145%
hi_card_long (~56B URLs)	89.63 ms	−2.2%	−13.9%	−5.0%	+7.0%	+83%

No change warranted:

• foldhash wins outright on low-cardinality categoricals — the shape that dominates real frequency/cardinality workloads — and ties (±3%) on timestamps and short high-card keys.
• gxhash's only win (−13.9%, long unique keys) is allocation-shadowed (each new key pays a to_vec) and priced in unsafe AES-SIMD that breaks plain cargo install builds — plus a +10% regression on the common case. Same narrow-win/common-regression pattern as previously rejected optimizations.
• ahash trails its designed successor foldhash on all four shapes.
• xxh3 via the Hasher trait is 2–3.5× slower everywhere — XXH3 is one-shot-oriented and the streaming per-key setup kills it on short keys; that interface is the only way a HashMap can use it, so it's disqualified regardless.
• fxhash's small long-key win costs per-instance seeding (HashDoS resistance) for parity elsewhere.

Full write-up with the reproducible harness source is now in the repo: qsv-stats/docs/perf/2026-08-hasher-shootout.md

So the pre-release checklist stands: algebraic ops (now unconditional, MSRV 1.98) were the one real win 1.98 offered; sort comparator guard re-verified green; hasher confirmed optimal. qsv-stats 0.55.0 is ready to cut.

🤖 Generated with Claude Code

[jqnatividad]

End-to-end measurement on the qsv-stats 0.55.0 bump (replacing the "low single digits" estimate) — commit 1534b62 adopts qsv-stats 0.55.0 (unconditional algebraic float ops) and bumps MSRV to 1.98.

Setup: hyperfine on qsv stats --everything --force over a 1M-row NYC 311 sample (514 MB, 41 cols), Apple M4 Max (16-core, aarch64), both binaries --release -F all_features with rustc 1.98.0.

path	qsv-stats 0.54.3	qsv-stats 0.55.0	speedup
indexed/parallel (20 runs)	358.3 ± 10.1 ms	352.4 ± 6.7 ms	1.02× ± 0.03
unindexed/sequential (10 runs)	2.011 ± 0.021 s	1.993 ± 0.048 s	1.01× ± 0.03

Low single digits end-to-end, as predicted — per-column the sort dominates the FP reduction work, and parsing/string columns/I-O dilute further. The adoption is worthwhile because it's essentially free and improves accuracy (~13× closer to Kahan at 1M elements in qsv-stats' own benchmarks).

Output stability: stats output on the 1M dataset is byte-identical between 0.54.3 and 0.55.0 at the default --round 4 — the ~1e-15 relative drift is fully absorbed. Full test suite green (995 unit + 3755 integration), no golden-value shifts, so no stats-cache cached-vs-fresh policy change is needed.

Still open (tracked separately, doesn't block closing):

• x86 verification: all numbers are aarch64/NEON. Expect the win to hold (it's accumulator-chain-breaking, not vector-width-dependent), and qsv-stats' 1M REDUCTION_PARALLEL_THRESHOLD crossover should be re-checked on x86.
• The side finding above (mul_add → libm call on baseline x86-64 cargo install builds) remains a separate decision.
• csvs_convert still pins qsv-stats 0.54.x, so both versions appear in Cargo.lock; harmless, but a fork bump would dedupe.

Closing per plan.