[fcostaoliveira]

Problem and change

check_evidence could mark a claim quotable even when it explicitly said that turbo and SMT states were unknown and runtime frequency was not measured. Conversely, equivalent wording such as Six runs, Clang18.1.3, or one pinned core could make a complete claim fail.

Recognise explicit unknown/unmeasured declarations, including coordinated fields, and keep nominal/advertised-only frequency incomplete. Preserve the existing result schema and conservative treatment of conflicting known/unknown declarations. Also recognise compact compiler versions, pinned cores/threads, and counted A/A or ABBA blocks. The checker remains a lexical metadata heuristic, not validation of the underlying measurement.

Example previously accepted, now drop_number:

AWS Graviton4, Arm Neoverse V2; one core pinned to CPU2; frequency 2.6 GHz nominal, turbo state unknown, SMT state unknown. Compiler Clang 18.1.3 -O3. Workload: B+ tree iterator benchmark, 100000 members. Compared with baseline revision A. 6 runs with median elapsed time, 7% faster. Frequency during the run was not measured.

These numbers are synthetic regression inputs, not benchmark results. A corresponding claim with stated runtime frequency and known states still passes.

Actual evaluation

The first commit (e150e17) adds the cases without changing the checker; the second fixes the checker and exercises the same cases through the MCP interface. Compared the original deb5a0b implementation and this branch against the same 36 authored cases:

Metric	Original	This PR
Cases passing verdict and field expectations	10/36	36/36
Incorrect quotable verdicts	18	0
Incorrect rejections of complete claims	8	0
Existing list retrieval eval	21/21	21/21

Positive controls include known disabled states, no-SMT hardware, estimated clocks with DVFS on, nominal plus measured frequency, unrelated unknown input metadata, and existing benchmark machine descriptions. Negative controls include unknown states before/after subjects, coordinated subjects, nominal-only clocks, conflicting declarations, absent clock metadata, and uncounted ABBA blocks.

Validation on Linux, Python 3.14.0:

• Full MCP suite: 211 passed, 2 skipped (32.44s); original suite: 139 passed, 2 skipped.
• All 36 cases pass through the in-process MCP client, checking structured fields and rendered verdicts.
• A separate real subprocess stdio MCP session passes all 36 rendered verdicts.
• cpu-perf --selftest passes.
• python3 misc/scripts/check_format.py passes.
• python3 misc/scripts/check_links.py exits 0: 260 OK, 37 blocked, 2 unreachable, 0 dead. No URLs changed; blocked existing URLs were not manually browser-verified in this run.

Upstream GitHub workflows currently report action_required with no jobs run. The results above are local validation, not an upstream CI pass.

The two skips require CPU_PERF_EVAL_DB pointing to a crawled source library. No source-library retrieval improvement or general NLP accuracy is claimed. The 36 cases are authored regressions, not a blinded or held-out evaluation.

Reproduce from an installed editable checkout:

python -m pytest misc/mcp/tests -q -rs -p no:cacheprovider
python misc/mcp/tests/eval_evidence.py
python misc/mcp/tests/eval_queries.py
cpu-perf --selftest

Run eval_evidence.py at e150e17 to reproduce the original 10/36 result, then at this branch's head for 36/36.

Scope and motivation

Found while using the MCP for standalone B+ tree performance investigations. This changes the MCP checker, tests and its documentation; it adds no curated source or performance claim. Typed metadata, library setup, and vendor-document resolution remain separate follow-ups.