RFC: Add `report` subcommand

[b-deam]

Summary

Today, users often run multiple races in order to have confidence in the results. They then either eyeball N summary reports, or use Elasticsearch aggregations against the raw samples in rally-metrics-* to get a better picture of the results and/or better understand the variability.

There is also no way to re-generate the summary report for a race after the fact. The report is printed once, at the end of a race.

Therefore, Rally should offer:

1. An easy way to re-generate the summary report for a single, existing race using the Rally CLI
2. An easy way to calculate statistics across N races using the Rally CLI
3. An easy way to report the variability/variance of tasks across those races using the Rally CLI

I propose the following:

1. An esrally report CLI subcommand that re-generates the summary report for one existing race.
2. When multiple race-ids are passed, esrally report generates a single summary by applying statistical aggregations across the samples of all the given races.
3. Reporting between-race variance through a Coefficient of Variation (CV) column in the multi-race esrally report output.

1. Re-generate the report for a single race

esrally report --race=<race-id> prints the summary report for an existing race, the same table you see at the end of esrally race.

By default the numbers come from the stored results (rally-results-*, or race.json for a file-based store), so this matches the original report exactly, is fast, and works even if the raw samples have since been deleted by ILM. If stored results are unavailable, Rally falls back to recomputing them from the raw samples in rally-metrics-* (the same path used for the multi-race case below).

CLI

esrally report --race=<race-id>

Behaviour

• Resolves the race the same way compare resolves --baseline / --contender. A missing race ID is an error.
• Prefers stored results, but falls back to recomputing from raw samples in rally-metrics-* when stored results are missing.
• Honours --report-format and --report-file (see options below).
• Read-only: it does not write new documents to the metrics store.
• No CV column is shown for a single race.

2. Aggregate results over N races

When more than one race is selected, esrally report prints one summary report computed across the samples of all the selected races, per task. The races must share a track and challenge.

CLI

# explicit race IDs
esrally report --race=<id1>,<id2>,<id3>

# or select races the same way as `list races`
esrally report --track=<track> [--challenge=<challenge>] [--user-tags=<k:v>] [--limit=N]

--race selects specific race-ids. If it is not set, races are selected with the same arguments as list races (--track, --challenge, --user-tags, --benchmark-name, --from-date, --to-date, --limit). If --race is set, the filter flags are ignored.

Options

Option	Behaviour
--race <ids>	One race ID, or a comma-separated list. If set, the filter flags are ignored.
--track / --challenge / --user-tags / --benchmark-name / --from-date / --to-date / --limit	Filter mode, same meaning as esrally list races.
--report-format markdown / csv	Same as compare.
--report-file	Also write the table to a file.

Behaviour

• Multi-race aggregation requires datastore.type = elasticsearch, as raw samples are not kept after an in-memory race.
• It requires the selected races to share the same track and challenge. A missing race ID is an error. Differing car or distribution-version is a warning, not an error.
• Per-task request metrics (throughput, latency, service_time, processing_time) are computed from the pooled raw samples in rally-metrics-* (e.g. terms on race-id, sample-type: normal only, extended_stats / percentiles). Warmup samples are excluded.
• Cluster-level, one-shot metrics (e.g. Cumulative indexing time, Total Young Gen GC time, Dataset size) are single values per race rather than sample streams. Their Aggregate value is the mean of the per-race values, and CV is computed over those same per-race values.
• Tasks are collected from the op_metrics of the selected races.
• Read-only: it does not write new documents to the metrics store.

3. Variance (CV)

For each row shown in the multi-race table, esrally report also reports the between-race Coefficient of Variation (CV). The CV summarises the repeatability of that result across the N races.

The Aggregate column and the CV column are computed differently, and this distinction matters:

• Aggregate is computed from the pooled normal samples of all selected races (for one-shot cluster metrics, the mean of the per-race values).
• CV shows how much a metric changed from race to race. Take that metric's value from each race (say the mean throughput from all 10 races), and CV is their standard deviation divided by their average, written as a percentage. A low CV (e.g. ~1-2%) means the races closely agreed. A high CV (e.g. ~10-20%) means the result swung a lot between runs. Note the average used here is the plain average of the 10 per-race values, which can differ slightly from the Aggregate column (that one is calculated from every raw sample in rally-metrics*).

CV is shown as - when it cannot be computed, e.g. fewer than two per-race values, or a mean of zero (e.g. an all-zero error rate).

Metric	Task	Aggregate	CV	Unit
Min Throughput	index-append	19118	6.1%	docs/s
Mean Throughput	index-append	20232	3.4%	docs/s
Median Throughput	index-append	20110	3.1%	docs/s
Max Throughput	index-append	21172	5.8%	docs/s
50th percentile latency	index-append	803.42	4.0%	ms
90th percentile latency	index-append	1913.70	7.2%	ms
99th percentile latency	index-append	3591.23	12.4%	ms
100th percentile latency	index-append	6642.97	18.1%	ms
50th percentile service time	index-append	612.10	3.8%	ms
90th percentile service time	index-append	1402.44	6.9%	ms
99th percentile service time	index-append	2810.02	11.8%	ms
error rate	index-append	0.00	-	%
Mean Throughput	term	50.12	2.1%	ops/s
50th percentile latency	term	2.13	5.5%	ms
90th percentile latency	term	2.81	8.0%	ms
99th percentile latency	term	4.72	14.6%	ms
100th percentile latency	term	5.15	22.0%	ms
error rate	term	0.00	-	%
Cumulative indexing time		55.80	2.8%	min
Total Young Gen GC time		4.12	9.5%	s
Dataset size		3.05	0.4%	GB

Out of scope

• Persisting an aggregated race as a new race / results document.
• A CV column on esrally compare.
• Comparing two groups of races against each other (each side aggregated, then diffed).

[pquentin]

Thanks for iterating! LGTM. Wondering if a JSON output would be useful as well.

[albin-george-kurian]

Hi @pquentin,
This Enhancement looks good. Can you assign this to me? I would like to work on it.