`--retry-failed-tests` leaves only the last attempt's report in the results directory, so TRX/CTRF/JUnit under-report the run

[Evangelink]

Describe the bug

When --retry-failed-tests is used, the report files left in the results directory (TRX, CTRF, JUnit, HTML) describe only the final attempt, not the run as a whole. The earlier attempts' reports are kept, but tucked away under Retries/<id>/<n>/.

Concretely, a 4-test suite that needed 4 attempts leaves a CTRF report in the results directory claiming the run had exactly one test.

The orchestrator moves the last attempt's files verbatim:

https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.Retry/RetrySummaryReporter.cs — MoveArtifactsAsync

Because each retry attempt is filtered down to the previously-failed tests, the last attempt's report is by construction the narrowest one — so the file a CI system picks up is the least representative of the run.

Version used

main (Microsoft.Testing.Extensions.Retry 2.4.0-dev, Microsoft.Testing.Platform 2.4.0-dev), reproduced locally from a fresh build.cmd -pack.

Steps To Reproduce

1. Create an MTP test app with 4 tests: 2 always pass, 1 fails only on its first run, 1 always fails.

2. Reference Microsoft.Testing.Extensions.CtrfReport (TRX and JUnit behave the same way).

3. Run:

   MyTests.exe --retry-failed-tests 3 --results-directory TR --report-ctrf
   

4. Inspect TR/*.ctrf.json — the file at the top level, not the ones under TR/Retries/.

Resulting layout:

TR\<name>.ctrf.json                       <- from attempt 4 only
TR\Retries\<id>\1\<name>.ctrf.json        <- attempt 1 (all 4 tests)
TR\Retries\<id>\2\<name>.ctrf.json        <- attempt 2 (2 tests)
TR\Retries\<id>\3\<name>.ctrf.json        <- attempt 3 (1 test)

Expected behavior

The report in the results directory should describe the whole run: 4 tests, with the recovered test marked as flaky and its earlier failed attempt(s) recorded.

For CTRF specifically that means summary.tests: 4 and "flaky": 1, with retries / retryAttempts[] populated on the test that recovered — the shape CtrfReportEngine.TestCollapsing.cs already defines.

Actual behavior

The top-level CTRF report describes only attempt 4:

{ "tests": 1, "passed": 0, "failed": 1, "skipped": 0, "pending": 0, "other": 0, "flaky": 0 }

Three of the four tests are absent, and the test that failed-then-passed is not represented at all — neither as passed, nor as flaky.

Additional context

The merging machinery already exists and is used elsewhere; it is simply not wired into the retry orchestrator:

• CtrfReportMerger.Merge(IReadOnlyList<string>) — pure JSON-level merge, re-derives summary counters from the merged tests[]
• TrxReportEngine.Merge.Orchestration.cs / .Merge.Summary.cs
• JUnitSuiteBuilder

So the fix is plausibly "merge the per-attempt reports instead of moving the last one" rather than new format work.

Points to decide while implementing:

• Collapse vs. concatenate. A naive concatenation would double-count tests that ran in several attempts. The merge needs the same per-uid collapse rule CTRF's CollapseAttempts uses (last attempt wins, earlier ones become retry attempts), otherwise summary.tests inflates.
• Formats without a retry concept. TRX and JUnit have no flaky/retry notion. Collapsing to the final outcome per test is probably right there, but it should be a deliberate decision rather than a side effect.
• Keep the per-attempt files. The Retries/<id>/<n>/ files are useful for debugging and should stay; this is about what the top-level report says.

Related: the console summary was fixed to report retried and flaky tests correctly in #5362 — so the terminal and the report files currently disagree about the same run.

[Youssef1313]

This is by-design at least for TRX.

[Evangelink]

Spec review — TRX, JUnit, CTRF

I went through the format documentation for the three report formats named here to work out, per format, whether the current behaviour is a spec problem, a product decision, or genuinely undefined. Summary first, details below.

Format	Does the format model retries?	Verdict
TRX	No	By design (as @Youssef1313 said). No change proposed.
JUnit XML	No — not in any portable dialect	Not a spec question. Undocumented behaviour → doc fixed; any change is a product decision, not a conformance fix.
CTRF	Yes — retries, retryAttempts[], flaky, and an explicit "one logical run = many physical executions" rule	The only format whose own spec has an opinion. Behaviour should change — but the exact output shape depends on ctrf-io/ctrf#58, which I opened.

---

TRX

TRX has no notion of an attempt, a rerun, or a flaky test. UnitTestResult carries one outcome, and InnerResults exists for data-driven rows, not reruns. Every mainstream consumer (Azure DevOps PublishTestResults in particular) counts one row as one test, so emitting the same test twice inflates the run. There is nothing in the format that the current behaviour contradicts. Agreed: by design, no change.

JUnit XML

There is no normative JUnit XML specification and no spec repo whose authors could be asked — the format is a family of dialects grown from Ant's junitreport. I re-checked the ones RFC 016 surveys:

• Jenkins / Surefire (the flavour we emit) — no rerun element.
• Maven Surefire 3.x — has <rerunFailure> / <rerunError> / <flakyFailure>, but they nest attempt outcomes inside the last <testcase>; parsers that don't know them mis-count. RFC 016 already rejected them, with reasons, and I still agree.
• testmoapp/junitxml — RFC 016 claimed a vendor-specific <flaky>; that is wrong, the reference documents no rerun/flaky element at all. I've corrected the table.
• pytest / xUnit.net --report-junit / NUnit 2 — no rerun support.
• JUnit 5 Open Test Reporting — first-class retries, but a completely different event-based schema that no CI service ingests.

So: no upstream spec to consult, and no portable way to say "this test failed and then passed". That leaves the JUnit behaviour purely as a product decision, with two honest options and no spec to break either way:

1. One row per logical test, final outcome — the merged file looks like an ordinary run of the suite (4 tests, 1 failure in the repro). History is lost, but CI gating and dashboards are correct.
2. One row per attempt, disambiguated with the [attempt N] + attempt-index/attempt-of convention RFC 016 already defines for in-process retries. History is kept, but failures counts every failed attempt, so a run rescued by retry still reads red in Jenkins/ADO/GitLab — which defeats the point of --retry-failed-tests.

My take is (1) if we change anything, precisely because (2) makes the retry feature counterproductive for the consumers that read this file. But it needs a maintainer call, not a conformance argument.

Meanwhile the RFC was factually misleading about what happens today:

Each attempt is a separate process and therefore produces its own JUnit XML file under the run's results directory. No special handling is required inside the report engine.

Attempts 1..N-1 land under <results>/Retries/<id>/<n>/ and only attempt N is moved to <results>/, so the file a CI job picks up is the narrowest one. I've rewritten that bullet to describe the real layout and link here.

CTRF

CTRF is the one format where the spec does take a position, and it takes it in three places:

• §2.1 Single Logical Run Representation — "A logical run MAY consist of multiple physical executions … Results from such executions MAY be merged into a single CTRF document."
• §2.7 Explicit Semantics Over Implicit Behavior — "test retries are represented explicitly as retry attempts".
• §9.20–9.22 + §11 — retries, retryAttempts[], flaky on the test object; §8.7 — summary.flaky; and the Terminology section defines a flaky test as one whose final status is passed after ≥1 failed attempt. That is exactly the run in the repro.
• §2.11 Immutable Report Artifacts — a merge "produces a new CTRF document and … assigns a different reportId", rather than modifying the originals. So keeping the per-attempt files under Retries/ and adding a new merged document at the top level is precisely the shape the spec describes.

So for CTRF the current output isn't invalid (attempt 4's document is a truthful document about attempt 4), but the file we present as the run's report is the one the spec gives us the vocabulary to write properly, and we don't. summary.tests: 4 / flaky: 1 with retryAttempts[] on the recovered test is representable today and is what a CTRF consumer expects.

Two implementation notes, both already hinted at in the description:

• CtrfReportMerger.Merge concatenates tests[]. Feeding it retry attempts would report the same test several times and inflate summary.tests; §4.9.1 also says producers "SHOULD NOT emit duplicate identity values within a single CTRF document when those duplicates would make correlation ambiguous". A retry merge therefore needs a collapse-by-identity mode. Note it must be an explicit mode, not the default: the merger is also used for cross-module merges, where MTP UIDs are only unique within an assembly, so collapsing by UID there would fuse same-named tests from different assemblies.
• Layering: Microsoft.Testing.Extensions.Retry only references Microsoft.Testing.Platform, so the orchestrator cannot merge a CTRF file itself. The natural seam is the existing IArtifactPostProcessor (RFC 018) — those services are already registered in the very service provider the orchestrator runs in (TestHostBuilder.BuildHostAsync → context.ServiceProvider.AddServices), so no relaunch is needed. It would need (a) a retry-specific entry point, since the shard merge and the retry merge have different semantics, ordered by attempt rather than by path, and (b) a CTRF post-processor, which doesn't exist yet — only TRX has one. TRX can simply opt out and keep today's behaviour.

Why I haven't implemented it yet

While reading the spec for the merge shape I hit a contradiction that decides what the merged document looks like, so I asked upstream first: ctrf-io/ctrf#58.

• §11 and the Terminology section say a retry attempt is a re-execution (attempt number > 1), but §11.1 says "Attempt number 1 represents the first execution", and the official examples/with-retries.json lists every attempt — 1 (failed), 2 (failed) and 3 (the final passing one) — while the test object's status is passed.
• §9.20 says retries SHOULD equal the number of entries in retryAttempts; the same example has "retries": 2 with three entries.
• Our engine currently emits attempts 1..N-1 (final outcome on the test object) — a third shape, which satisfies §9.20 but not the example.

Whichever answer we get applies to both the in-process path (CtrfReportEngine.CollapseAttempts, which may need adjusting) and the orchestrated path, so it is worth settling before writing the merge. I also asked whether the merged document and the per-attempt documents should share a runId.

Proposal

Once #58 comes back:

1. Add CtrfArtifactPostProcessor (also useful on its own for multi-module dotnet test, where CTRF has no post-processor today).
2. Give CtrfReportMerger an explicit collapse-by-identity mode: last attempt wins, earlier ones become retryAttempts[], flaky set per the spec rule, summary re-derived from the collapsed tests[].
3. Let the retry orchestrator run post-processors over the per-attempt artifacts in attempt order after the final attempt, writing the consolidated report into the results directory. Per-attempt files under Retries/ stay put.
4. TRX opts out (by design). JUnit follows whichever option above is chosen.

Happy to take that on — just want a nod on the JUnit option and on the orchestrator seam before writing it, since both are product calls rather than conformance fixes.