moarstats: `--pct-thresholds` silently yields 0 for any percentile not in the default `--percentile-list`

[jqnatividad]

Summary

qsv moarstats --use-percentiles --pct-thresholds <lower>,<upper> silently produces wrong winsorized/trimmed statistics — 0 in the worst case — whenever the requested thresholds are not among the percentiles stats actually computed. It exits 0 with no warning.

The cause is that moarstats invokes stats itself without forwarding a matching --percentile-list, so stats always computes the default 5,10,40,60,90,95. moarstats then looks up the requested threshold as a key in the resulting percentiles cell. Any threshold outside that default is simply not found.

Reproduction

printf 'n\n' > d.csv && seq 1 100 >> d.csv
qsv moarstats --use-percentiles --pct-thresholds 7,93 d.csv
# inspect winsorized_mean_7pct in d.stats.csv

Column n is 1..100, so any winsorized mean must be near 50.5.

--pct-thresholds	column emitted	value	correct?
5,95	winsorized_mean_5pct	50.45	✅ (5 and 95 are in the default list)
10,90	winsorized_mean_10pct	50.4	✅ (both in the default list)
0.5,95.5	winsorized_mean_0pct	50.35	❌ lower bound not applied (truncates to 0, absent)
7,93	winsorized_mean_7pct	0	❌ neither bound found

7,93 are plain integers — nothing to do with fractional input or with the 0 edge case. Any pair outside 5,10,40,60,90,95 reproduces it. When one bound resolves and the other does not, the result is a partially-winsorized value (wrong but plausible-looking); when neither resolves, it collapses to 0.

Confirmation of the cause

Pre-building the stats cache with a matching percentile list fixes it, with no other change to the command:

printf 'n\n' > d.csv && seq 1 100 >> d.csv

# without a matching cache
qsv moarstats --use-percentiles --pct-thresholds 7,93 d.csv
#   percentiles      = 5: 5|10: 10|40: 40|60: 60|90: 90|95: 95
#   winsorized_mean_7pct = 0

# with a matching cache
qsv stats --everything --percentiles --percentile-list 7,93 --output /dev/null d.csv
qsv moarstats --use-percentiles --pct-thresholds 7,93 d.csv
#   percentiles      = 7: 8|93: 93
#   winsorized_mean_7pct = 50.5

Suggested fix

Forward the thresholds to the stats run that moarstats performs — i.e. pass --percentile-list containing (at least) the two requested percentiles — so the values moarstats looks up are guaranteed to exist.

Validating harder at the front instead would be the wrong fix: it would reject legitimate requests like 7,93 that stats is perfectly capable of computing.

Whatever the fix, the "requested percentile not present in the cell" path should not stay silent. Today it returns 0/partial at exit 0, which is indistinguishable from a real result.

Notes

• Pre-existing; not introduced by the Batch C percentile work in fix(stats): wrong-results cluster (Batch C of the stats.rs review) #4443. That change truncates fractional thresholds to the integer percentile stats computes, which is correct and unrelated — the integer pair 7,93 fails identically.
• Reachable only via --use-percentiles together with a non-default --pct-thresholds.
• An existing cache from a prior stats run with a different --percentile-list will change the outcome, so reproduce in a clean directory.
• Found while reviewing the merge of fix(stats): wrong-results cluster (Batch C of the stats.rs review) #4443 into fix(stats): date renderings are no longer coerced to 0.0 in the stats cache #4441; version qsv 22.0.1, aarch64-apple-darwin.