You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Every prior attempt is dropped there and never becomes a TestNodeUpdateMessage. As a result a test decorated with [Retry(n)] that fails and then passes is reported as an ordinary passing test — no attempt count, no flaky marker, in the terminal or in any report format.
The RetryResult XML doc already anticipates this:
The other results are currently not used, but may be used in the future for tooling to show the state of the failed attempts.
This issue is that follow-up.
Background and Motivation
Two concrete consequences, both verified against main with a [Retry(3)] test that fails on its first attempt and passes on its second:
1. The terminal shows nothing.
Test run summary: Passed! - RetryDemo.dll (net8.0|x64)
total: 1
failed: 0
succeeded: 1
skipped: 0
duration: 378ms
with no retries count and no retryAttempts[] array on the test entry.
That second point matters beyond cosmetics: Microsoft.Testing.Extensions.CtrfReport already implements the correct flaky rule in CtrfReportEngine.TestCollapsing.cs (CollapseAttempts + IsFlaky), and CtrfReportEngine.JsonTestWriter already knows how to emit retries and retryAttempts[]. Because no prior attempt ever reaches the message bus, that entire code path is unreachable for [Retry] and the emitted report is silently wrong for consumers that key off flaky.
The out-of-process --retry-failed-tests orchestrator now reports flaky tests by name (#5362), which makes the asymmetry more visible: the same flakiness is surfaced when the whole host is relaunched, but hidden when [Retry] handles it in-process.
Proposed Feature
Report each [Retry] attempt as a TestNodeUpdateMessage carrying its 1-based attempt number (a new RetryAttemptProperty, or an equivalent already-serialized property).
The consuming side largely exists already:
TestProgressState.ReportGenericTestResult already implements "same uid, higher attempt → un-count the previous attempt, count it as retried, remember the earlier failure". The flaky: / retried: summary lines and the Flaky tests: section added in show summary of retried and flaky tests #5362 would therefore light up for [Retry] with no extra rendering work.
CTRF's CollapseAttempts / IsFlaky would start producing correct flaky, retries and retryAttempts[] values.
Things to work through
TRX/JUnit shape. These formats would gain rows for prior attempts unless they go through the same per-uid collapse rule. This is the main risk and should be decided before implementing.
Attempt granularity for data-driven tests. Attribution is per TestNodeUid, so a folded (TestDataSourceUnfoldingStrategy.Fold) data source reports one entry regardless of how many rows ran — the same documented limitation as show summary of retried and flaky tests #5362.
Interaction with --retry-failed-tests. Attempt numbers are already used by the out-of-process orchestrator via TESTINGPLATFORM_DOTNETTEST_ATTEMPTNUMBER. When both are enabled the counts multiply (as RetryAttribute's docs note), so the two attempt notions need to compose rather than collide.
Alternative Designs
Report only a summary property (e.g. "this test was retried N times, first outcome X") rather than a full message per attempt. Cheaper and avoids the TRX row-count question entirely, but loses per-attempt error messages and durations, and would not populate CTRF's retryAttempts[].
Leave the message stream alone and expose the attempts through a side channel consumed only by the terminal reporter. Avoids affecting report formats, but then the reports stay wrong, which is the more important half of the problem.
Do nothing and document it. Rejected: the CTRF flaky field is machine-readable output that is currently incorrect rather than merely absent.
Follow-up split out of #5362; the out-of-process half is implemented, this is the in-process half.
Summary
RetryBaseAttribute.ExecuteAsyncreturns aRetryResultholding every attempt, but the adapter keeps only the last one:https://github.com/microsoft/testfx/blob/main/src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs#L246
Every prior attempt is dropped there and never becomes a
TestNodeUpdateMessage. As a result a test decorated with[Retry(n)]that fails and then passes is reported as an ordinary passing test — no attempt count, no flaky marker, in the terminal or in any report format.The
RetryResultXML doc already anticipates this:This issue is that follow-up.
Background and Motivation
Two concrete consequences, both verified against
mainwith a[Retry(3)]test that fails on its first attempt and passes on its second:1. The terminal shows nothing.
2. The CTRF report claims the test is not flaky.
{ "tests": 1, "passed": 1, "failed": 0, "skipped": 0, "flaky": 0 }with no
retriescount and noretryAttempts[]array on the test entry.That second point matters beyond cosmetics:
Microsoft.Testing.Extensions.CtrfReportalready implements the correct flaky rule inCtrfReportEngine.TestCollapsing.cs(CollapseAttempts+IsFlaky), andCtrfReportEngine.JsonTestWriteralready knows how to emitretriesandretryAttempts[]. Because no prior attempt ever reaches the message bus, that entire code path is unreachable for[Retry]and the emitted report is silently wrong for consumers that key offflaky.The out-of-process
--retry-failed-testsorchestrator now reports flaky tests by name (#5362), which makes the asymmetry more visible: the same flakiness is surfaced when the whole host is relaunched, but hidden when[Retry]handles it in-process.Proposed Feature
Report each
[Retry]attempt as aTestNodeUpdateMessagecarrying its 1-based attempt number (a newRetryAttemptProperty, or an equivalent already-serialized property).The consuming side largely exists already:
TestProgressState.ReportGenericTestResultalready implements "same uid, higher attempt → un-count the previous attempt, count it as retried, remember the earlier failure". Theflaky:/retried:summary lines and theFlaky tests:section added in show summary of retried and flaky tests #5362 would therefore light up for[Retry]with no extra rendering work.CollapseAttempts/IsFlakywould start producing correctflaky,retriesandretryAttempts[]values.Things to work through
TestNodeUid, so a folded (TestDataSourceUnfoldingStrategy.Fold) data source reports one entry regardless of how many rows ran — the same documented limitation as show summary of retried and flaky tests #5362.--retry-failed-tests. Attempt numbers are already used by the out-of-process orchestrator viaTESTINGPLATFORM_DOTNETTEST_ATTEMPTNUMBER. When both are enabled the counts multiply (asRetryAttribute's docs note), so the two attempt notions need to compose rather than collide.Alternative Designs
retryAttempts[].flakyfield is machine-readable output that is currently incorrect rather than merely absent.Follow-up split out of #5362; the out-of-process half is implemented, this is the in-process half.