Repository navigation
Expose expected-vs-emitted check coverage per scenario #451
Description
Activity
Reproduced the passing thin shadow outside mcpkit, on a plain server run, in one command.
Measured 2026-09-05 with
@modelcontextprotocol/conformance@0.2.0-alpha.11, source read at7160fc8f.Setup: a stateful 2025-11-25 SDK server, run under
--requirements 2026-07-28. The transport answers every request with-32000 Bad Request: Server not initialized. 36 of the 37 scored scenarios fail. The one that passes isinput-required-result-validate-input, 3 of 3 green:SUCCESS | sep-2322-validate-input-responses SUCCESS | sep-2322-error-on-protocol-error SUCCESS | wire-schema-validThose checks expect a JSON-RPC error, the server returns one for an unrelated reason, and the scenario reads green. No setup failure is involved, which is the harder case you describe.
A second shape the counts hide:
tasks-status-notificationsreports0 passed, 0 failedin both of my runs. Its single check is SKIPPED, documented in AGENTS.md as intended, but a zero denominator still renders as a tick downstream.A third, on disk rather than in the summary: a scenario that throws creates its output directory (
src/runner/server.ts:44-47) but never writeschecks.json(:105), since the throw happens atscenario.run(:101). The synthetic FAILURE atsrc/index.ts:682-697never reaches the file. One 50 scenario run left 50 directories and 49 files.To read results programmatically we compare the
checks.jsonset againstrequirements/<revision>.yaml: a required scenario with no file counts as crashed, one whose checks contain neither SUCCESS nor FAILURE counts as empty rather than passed. That is a consumer side workaround for the signal this issue asks for upstream.One thing that would help file consumers without settling the expected set question: write
checks.jsoneven when a scenario throws, so the emitted count is never silently zero.This analysis was done with the help of an AI assistant and verified by running the suite; this comment was drafted with AI assistance.
Reproduction harness: https://github.com/AmirK-S/mcp-migrate-delta
I reproduced the missing checks.json exception subcase reported by AmirK-S, at conformance commit a983ba9, using SDK 1.29.0 on a loopback server and Node 22.16.0 on Windows.
The source CLI and runner files were unchanged; a test-only adapter injected controlled scenarios into the in-memory registry. Success, explicit FAILURE and timeout wrote checks.json. A synchronous throw before connection, and a rejection after a real SDK exchange, exited 1 but left their result directories without checks.json. The suite path correctly synthesized FAILURE in memory: five executed scenarios produced five directories but only three checks.json files. A separately requested inapplicable scenario exited 0 with an explicit SKIPPED message. Stock built-CLI controls are recorded separately.
This supports the persistence subproblem only, not full expected-versus-emitted coverage or a claim that the CLI silently passes crashes. I have not prepared a product patch, and I recognize the existing work discussed by panyam and AmirK-S. Would a small reproduction/persistence follow-up be useful, or is it already covered by ongoing work? The artifacts were prepared with AI assistance; cross-SDK validation remains outstanding before any shared-runner PR.
@mohammedmessaoudene yes, the persistence part is worth doing on its own and nobody is on it. Writing
checks.jsonwhenscenario.runthrows closes the 50 directories / 49 files gap AmirK-S measured: the synthetic FAILURE fromsrc/index.tsexists in memory and never reaches disk. It does not need the expected-set question settled first. Go ahead if you want it, I'll review the PR.@AmirK-S
input-required-result-validate-inputgoing 3/3 green on a server that answers everything with-32000 Server not initializedis the case I could not show outside mcpkit: the check expects a JSON-RPC error and gets one for an unrelated reason. That one and the0 passed, 0 failedscenario are what the expected-vs-emitted count in this issue has to catch, so I'm keeping this open for that part.@rinaldofesta thanks, agreed that the count is what this issue stays open for.
The two scenario-level cases can be closed directly in the meantime, so I opened #537. Against a server that answers everything with
-32000,--requirements 2026-07-28onmaingives two fully green scenarios:input-required-result-validate-inputandinput-required-result-unsupported-methods, which the rejected probes never exercise. With a control call on each, reported as untestable per #248, that goes to zero.One data point for the count design: the same server also turns individual checks green inside scenarios that stay red overall, for example
sep-2575-http-server-meta-invalid-400inserver-stateless, since the server returns 400 to everything. A per-scenario count would not show those; a control request does, asdns-rebinding-protectionalready has withlocalhost-host-valid-accepted.Drafted with AI assistance.
@pcarleton, #248 flagged "a scenario knowing all its checks" as the bigger change. Does deriving the expected set from the traceability run work for you as the baseline? If so I'm happy to do the PR: per-scenario IDs in the manifest, then
emitted/expectedin the summary and in machine-readable output, with tests for the setup-failure and thin-shadow cases. We can then retire our snapshot guard once it is done.Happy to share numbers again from mcpkit's conformance reports and runs etc.
A scenario can emit only a thin subset of its checks while the reported denominator makes that subset look complete.
The concrete case from mcpkit:
auth/authorization-server-migrationreported2 passed, 1 failedfor months. The initial wire handshake failed, so roughly 76 other checks never emitted. PR #327 makes the setup failure explicit, but the result still cannot show how much of the scenario did not run.The harder case has no setup failure: setup succeeds, only accept-path checkpoints emit, and the thin scenario result passes.
Add a per-scenario expected-vs-emitted coverage signal to the upstream results. It should be visible in both terminal summaries and machine-readable output. A count is enough to expose the gap; stable IDs would also identify the missing checks.
Today
ScenarioandClientScenariodo not declare their expected check IDs, and some checks are dynamic or version-gated. The implementation will need a stable expected set or another baseline that excludes checks which are not applicable to the selected spec version or extension.Acceptance criteria:
@panyam offered to implement this in #327.
Prior art: panyam/mcpkit#1115