stats: re-enable parallel processing for special-format inputs (#4462)

[jqnatividad]

Closes #4462.

stats refused to autoindex a special-format input (.gz/.zip/.parquet/.jsonl/…) and so always ran sequentially. That guard was the narrow fix in #4445 for a real crash: parallel_stats built a fresh Config per worker, each of which resolved its own converted temp, found no index beside it, and panicked — silently, since a panicking pool worker only unwinds its own thread, leaving a headers-only stats file at exit 0.

This adopts the pattern #4459 established across split/search/searchset/pragmastat and #4466 made safe: share one resolved Config. Clones share the Arc<OnceLock> holding the conversion, so every worker sees the same temp and the same sibling index.

Changes

• sequential_stats, parallel_stats and process_headers_with_weight_exclusion now take &Config; parallel workers get a clone. Exactly one Config is constructed per run, so the invariant is greppable.
• All three is_special_format() guards removed. The memcheck index fallback mattered most — it fires precisely when the input is too large to process sequentially, so excluding compressed inputs left them with no index escape hatch at all.
• Autoindex cleanup now resolves the index beside the path that was actually indexed (the temp), not the compressed source — otherwise every such run logged a spurious Could not remove index file.
• Config::resolve_converted logs one line per conversion, inside the get_or_init closure so it fires exactly once per Config family. This makes Config-rebuilding regressions countable instead of invisible.

On the #4446 repro: nchunks=17 where master logged none, byte-identical to the uncompressed input, and conversions dropped 3 → 1.

Two hazards found along the way

A silent wrong-results trap. stdin delimiter inference sets args.flag_delimiter after rconfig is built, and relied on the compute paths rebuilding their own Config to pick it up. Sharing the pre-built one parsed a TSV as a single comma-delimited column — wrong stats, exit 0, no warning. Re-applied explicitly; stats_infers_stdin_delimiter pins it.

A stale claim in the comments. An earlier note asserted moarstats still rebuilds a Config per worker from the original path. #4464 already fixed that — compute_outliers_and_kga and compute_all_bivariatestats are both handed read_input_path, already resolved by their caller. Left as written, the comment would have sent a maintainer to fix a bug that does not exist. Corrected, and the second valid shape (resolve once, pass the resolved path) is now named alongside the first (share the resolved Config).

Tests

stats_autoindex_is_skipped_for_special_format_inputs is reworked into stats_parallelizes_special_format_inputs. It keeps the valuable half — .zip stats must equal the same data uncompressed — and drops one assertion that was vacuous: no index was ever written beside the compressed source, so it could not fail either way.

Output cannot distinguish sequential from parallel (identical by design, which is the point), so the discriminating evidence is the log: a nchunks= line plus exactly one conversion. Coverage spans both autoindex routes (--cache-threshold and QSV_AUTOINDEX_SIZE) and the sequential path.

All seven code changes were mutation-verified: each reverted individually, with the corresponding assertion confirmed to fail. Reverting the worker change reproduces the original crash. An early mutation pass passed when it should not have, which is what surfaced the uncovered QSV_AUTOINDEX_SIZE route.

Both tests are also pinned against an ambient QSV_AUTOINDEX_SIZE/QSV_DEFAULT_DELIMITER — Workdir::command inherits the environment — and that pinning was itself verified load-bearing.

Green locally: all_features 992 unit + 3748 integration, lite 132 + 2251, clippy clean on both. The .zip fixture is deliberate so the new tests run under -F lite too.

Risk

The concurrency delta is small and one-directional. Workers now carry a non-zero autoindex_size where they previously inherited the env value, so a TOCTOU-deleted index self-heals instead of panicking. The staleness recheck cannot flip — the temp is written once and never touched, so its mtime cannot exceed the index's regardless of machine speed. That is reasoning rather than proof, and a green local suite is not evidence about CI, so macOS and qsvlite runners are the real check.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric	Results
Complexity	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}