fix(perf): leave unrecorded optional metrics out of perf detail rows

[RizgarOzan]

Fixes #1786

Since #1744, _summary_values returns None for avg_steady_itl, the PD handoff metrics and their p99s when a run didn't record them. PerfSummaryRow.values is Dict[str, float], so json_response fails validation and /api/v1/perf/detail returns 500 for every normal (non-PD) run. The HTML report isn't affected because it only formats the columns it shows.

build_run_detail now drops None values from each row before building the response. The contract stays Dict[str, float], and the dashboard already treats a missing key as empty (row.values[key] ?? '', formatMetric placeholder), which also covers a sweep where only some runs have PD data.

How I tested:

• tests/perf/test_perf_archive.py: the existing test_detail_returns_summary_fields fails on main with this 500 (CI doesn't run this file). Added an assertion for the missing key and a test with one PD run and one non-PD run. Both red on main, 15/15 green with the fix.
• pytest tests/service tests/report tests/perf/test_perf_report_loader.py tests/perf/test_perf_visualizer.py tests/perf/test_rich_display.py: passing. tests/perf/test_stream_metrics.py has 6 failures on my Windows machine, same 6 on main.
• test_ci_lite (mock LLM) passes, pre-commit clean.

[Yunnglin]

Thanks for the careful diagnosis and the detailed write-up — your root-cause analysis of #1786 is exactly right: _summary_values unconditionally emits the Optional[float] PD/steady-ITL metrics, and PerfSummaryRow.values: Dict[str, float] turns those Nones into a 500.

Unfortunately we landed a fix for the same bug in #1790 (commit c8dc127) before this PR arrived, and it takes a different approach that makes this one hard to rebase:

• The fix lives in build_summary_table (evalscope/perf/utils/report/summary.py) rather than in build_run_detail, so the pruned column set becomes the single source of truth and both values and sample_counts are projected onto it. That also keeps the HTML report path (generate_report.py) consistent with the REST path, since both share the same producer.
• The pruning condition changed from any to all: an optional column survives only when every run measured it.

Two consequences for this PR:

1. After a merge, the 4 added lines in perf_archive.py become a provable no-op, and tests/perf/test_perf_archive.py conflicts textually with the tests added in fix(perf): stop perf/detail 500 for runs without PD metrics #1790.
2. The semantics are mutually exclusive. Your test_detail_with_optional_metric_on_some_runs asserts the PD column is kept when one run measured it; main now asserts the opposite in test_detail_prunes_pd_column_when_partially_measured.

Beyond the overlap, there are two issues worth recording in case they come up again:

• RunData.get_p99() → PercentileResult.get_p() returns 0.0 (not None) when the percentile row is missing, so filtering on is not None strips avg_pd_handoff_latency but leaves a fabricated p99_pd_handoff_latency = 0.0. The dashboard's toMetricMap treats any finite number as a real observation, so that 0.0 would enter delta computation and produce an improvement/regression verdict for a metric that was never measured.
• Stripping values but not sample_counts de-syncs the two key sets. getSampleCount looks up by column key, so a kept column whose value was removed still reports n=2 → classifySampleSize marks it 'critical', i.e. the UI downgrades the metric as "too few samples" when the truth is "never measured".

So we're going to close this as superseded — nothing here is a judgement on the quality of your investigation, which was spot on.

If you'd like to keep contributing in this area, a genuinely uncovered case would be a regression test asserting that the p99 siblings of an unmeasured optional metric never leak a fabricated 0.0 (under main semantics: assert both the avg and p99 PD columns are pruned). Happy to review that.