stats: skip the row-count pre-pass on unindexed files (perf; qsvlite is the main beneficiary)

[jqnatividad]

This is the performance half of the old R17 finding. The correctness half — seeding
RECORD_COUNT from a separate pre-pass, which corrupted per-record denominators — has been
fixed separately; this issue is only about the remaining wasted scan.

What remains

On the unindexed path, run() still calls util::count_rows(&rconfig) before the compute
pass. It is now used only as a capacity hint for the per-column accumulators. The file
is therefore read twice.

Measurement

Release binaries, warm page cache, 539 MB / 1M-row NYC 311 sample, unindexed, cache disabled.
qsv count is the cost that would be removed; it is the ceiling on the win.

build	mode	pre-pass	total	share
qsv (polars)	plain	0.07s	0.87s	~8%
qsv (polars)	-E	0.07s	2.09s	~3%
qsvlite	plain	0.33s	1.12s	~30%
qsvlite	-E	0.33s	2.51s	~13%

qsvlite is the real beneficiary. With polars, the pre-pass is a mem-mapped newline scan;
without it (count_with_csv_reader) it is a genuine CSV parse, hence the 4-5x difference.

Note also that a cold cache does not double disk I/O — the second read is served from the
page cache — so the win is roughly the warm numbers above except for files larger than RAM.

Why it is not a one-liner

1. expected_rows is not a lazily-growing hint — it is a real upfront allocation.
   It reaches stats::Unsorted::with_capacity(record_count) and Vec::with_capacity(record_count)
   in Stats::new, plus Frequencies::with_capacity((record_count / 10).clamp(16, 65_536)).
   Today the hint is exactly right. Replacing it with a filesize/avg-record-size estimate makes
   overshoot cost real RSS; only undershoot is free (organic growth).

2. It needs an estimator decision (the old U6 lead). calculate_avg_record_size is
   re-implemented inline elsewhere with a divergent .max(1024) clamp. Removing the pre-pass
   forces a choice of one implementation.

3. util::ROW_COUNT becomes live again. util::count_rows() sets it as a side effect, and
   it feeds util::optimal_batch_size. stats does not use optimal_batch_size, so this is
   currently a non-issue — but that is only true while the call remains. Removing the call
   makes this check load-bearing; re-verify it rather than inheriting an "already checked" note.

Not in scope

The indexed path is unaffected — it gets its count from the index instantaneously, with no
extra scan.

[jqnatividad]

Investigation findings

The hint is consumed by nothing at all in plain mode

Tracing capacity_hint → sequential_stats → compute/compute_pipelined → new_stats → Stats::new: the value reaches exactly four allocations — Unsorted::with_capacity / weighted Vec::with_capacity (exact quantiles only) and Frequencies::with_capacity (clamped to 65,536) / weighted HashMap::with_capacity (exact modes/cardinality only). In plain qsv stats (no -E/--median/--quartiles/--mad/--percentiles/--mode/--cardinality) none of these are active — the pre-pass was 100% waste in the default mode, which is why the "plain" rows of the table capture the full ceiling below.

Complication 1 (allocation reality) — the overshoot/undershoot asymmetry is inverted for the Vec case

Measured, not theorized: a raw filesize/avg estimate came out 99.4% of actual on the 1M-row NYC file — and that 0.6% undershoot cost +12 MB RSS, because every dense column's Unsorted Vec realloc-doubles the moment the estimate is exceeded. Overshoot is nearly free: Vec::with_capacity reserves without touching pages, so an unfilled tail costs virtual space, not RSS. (The exact count was already an over-reservation for sparse columns anyway.) So the estimate gets a +5% pad; with it, RSS is back to +0.3% vs the exact pre-pass. Frequencies keeps its existing 65,536 clamp; the eager-bucket concern stays covered.

Complication 2 (estimator decision) — neither existing helper is the right tool

calculate_avg_record_size and the util.rs inline variant both estimate the in-memory footprint per record (which_stats-aware) for chunk sizing — not on-disk bytes. Row-count estimation needs on-disk bytes, so the fix uses a new position-based sampler: read up to 1,000 records tracking rdr.position().byte(), giving the exact on-disk average (quotes, delimiters, terminators included — no fudge factors). If EOF arrives inside the sample, that count is exact. Skipped blank lines inflate the average → deflate the estimate, and errors degrade to a hint of 0 (an unreadable input fails moments later in the compute pass with the real error).

Complication 3 (ROW_COUNT liveness) — re-verified, dead

util::get_stats_records launches stats as a subprocess (std::process::Command), so the ROW_COUNT side effect dies with the stats process; the parent-side optimal_batch_size callers (apply, datefmt, template, validate, tojsonl) never see it. Within the stats process itself, nothing calls optimal_batch_size. Also checked moarstats: its count_rows fallback is a correctness comparison (cardinality == rowcount for all-unique detection), not a hint — correctly out of scope; an estimate must not be substituted there.

Measured result (release qsvlite, the same 539 MB / 1M-row NYC file, unindexed, cache disabled, warm)

mode	before	after	win	issue's ceiling
plain	1.169 s	0.813 s	1.44× (−30%)	~30% — full capture
-E	2.66 s	2.21 s	~1.20× (−17%)	~13%

Outputs byte-identical in both modes; max RSS 645.8 → 647.9 MB (+0.3%). The -E win exceeds the ceiling because the pre-pass it removes also carried per-record overhead the sampler doesn't.

PR incoming.