Skip to content

MSTest [Retry] attempts are discarded before they reach the platform, so in-process retries are invisible everywhere #10292

Description

@Evangelink

Summary

RetryBaseAttribute.ExecuteAsync returns a RetryResult holding every attempt, but the adapter keeps only the last one:

https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs#L246

result = retryResult.TryGetLast() ?? throw ApplicationStateGuard.Unreachable();

Every prior attempt is dropped there and never becomes a TestNodeUpdateMessage. As a result a test decorated with [Retry(n)] that fails and then passes is reported as an ordinary passing test — no attempt count, no flaky marker, in the terminal or in any report format.

The RetryResult XML doc already anticipates this:

The other results are currently not used, but may be used in the future for tooling to show the state of the failed attempts.

This issue is that follow-up.

Background and Motivation

Two concrete consequences, both verified against main with a [Retry(3)] test that fails on its first attempt and passes on its second:

1. The terminal shows nothing.

Test run summary: Passed! - RetryDemo.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 378ms

2. The CTRF report claims the test is not flaky.

{ "tests": 1, "passed": 1, "failed": 0, "skipped": 0, "flaky": 0 }

with no retries count and no retryAttempts[] array on the test entry.

That second point matters beyond cosmetics: Microsoft.Testing.Extensions.CtrfReport already implements the correct flaky rule in CtrfReportEngine.TestCollapsing.cs (CollapseAttempts + IsFlaky), and CtrfReportEngine.JsonTestWriter already knows how to emit retries and retryAttempts[]. Because no prior attempt ever reaches the message bus, that entire code path is unreachable for [Retry] and the emitted report is silently wrong for consumers that key off flaky.

The out-of-process --retry-failed-tests orchestrator now reports flaky tests by name (#5362), which makes the asymmetry more visible: the same flakiness is surfaced when the whole host is relaunched, but hidden when [Retry] handles it in-process.

Proposed Feature

Report each [Retry] attempt as a TestNodeUpdateMessage carrying its 1-based attempt number (a new RetryAttemptProperty, or an equivalent already-serialized property).

The consuming side largely exists already:

  • TestProgressState.ReportGenericTestResult already implements "same uid, higher attempt → un-count the previous attempt, count it as retried, remember the earlier failure". The flaky: / retried: summary lines and the Flaky tests: section added in show summary of retried and flaky tests #5362 would therefore light up for [Retry] with no extra rendering work.
  • CTRF's CollapseAttempts / IsFlaky would start producing correct flaky, retries and retryAttempts[] values.

Things to work through

  • TRX/JUnit shape. These formats would gain rows for prior attempts unless they go through the same per-uid collapse rule. This is the main risk and should be decided before implementing.
  • Attempt granularity for data-driven tests. Attribution is per TestNodeUid, so a folded (TestDataSourceUnfoldingStrategy.Fold) data source reports one entry regardless of how many rows ran — the same documented limitation as show summary of retried and flaky tests #5362.
  • Interaction with --retry-failed-tests. Attempt numbers are already used by the out-of-process orchestrator via TESTINGPLATFORM_DOTNETTEST_ATTEMPTNUMBER. When both are enabled the counts multiply (as RetryAttribute's docs note), so the two attempt notions need to compose rather than collide.

Alternative Designs

  • Report only a summary property (e.g. "this test was retried N times, first outcome X") rather than a full message per attempt. Cheaper and avoids the TRX row-count question entirely, but loses per-attempt error messages and durations, and would not populate CTRF's retryAttempts[].
  • Leave the message stream alone and expose the attempts through a side channel consumed only by the terminal reporter. Avoids affecting report formats, but then the reports stay wrong, which is the more important half of the problem.
  • Do nothing and document it. Rejected: the CTRF flaky field is machine-readable output that is currently incorrect rather than merely absent.

Follow-up split out of #5362; the out-of-process half is implemented, this is the in-process half.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/mstestMSTest framework (not analyzers or assertions).area/mtpMicrosoft.Testing.Platform core library.needs/triageNeeds triage by a maintainer.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions