Add on-device retry rule for flaky instrumented tests, Fixes AB#3678341

[p3dr0rv]

What

Adds a decoupled JUnit4 retry mechanism for instrumented (androidTest) tests. The Gradle org.gradle.test-retry plugin only retries JVM/Robolectric unit tests (tasks.withType(Test)); it does nothing for connectedAndroidTest.

Changes

• FlakyTestRetryRule (TestRule) + @RetryFlakyTest annotation in testutils. Opt-in, honours AssumptionViolatedException (skips aren't retried). Names are intentionally distinct from the UI-automation RetryTestRule/RetryOnFailure to prevent silent mis-imports.
• Wire testutils into common androidTest with transitive = false (pulls only the rule classes; avoids dragging testutils' transitive graph e.g. a second BouncyCastle line).
• RetryRuleValidationTest — sanity check that the rule re-runs and passes on a later attempt.
• pr-common-instrumented.yml — PR-validation pipeline running common:connectedLocalDebugAndroidTest on an emulator.

AB#3678341

https://identitydivision.visualstudio.com/Engineering/_build/results?buildId=1665439&view=results

Usage

@get:Rule val flakyTestRetryRule = FlakyTestRetryRule()

@Test
@RetryFlakyTest(retryCount = 2) // up to 3 attempts
fun someFlakyTest() { ... }
```n
> Retry is a mitigation, not a fix — it stays opt-in so flaky tests remain visible and get fixed.

[github-actions]

❌ Work item link check failed. Description does not contain AB#{ID}.

Click here to Learn more.

[Copilot]

Pull request overview

Adds an opt-in, on-device retry mechanism for flaky instrumented (androidTest) JUnit4 tests, and wires it into Common’s instrumented test suite along with a new PR-validation pipeline to run common:connectedLocalDebugAndroidTest on an emulator.

Changes:

• Introduces FlakyTestRetryRule (TestRule) and @RetryFlakyTest annotation in testutils for retrying annotated instrumented tests.
• Wires :testutils into common’s androidTestImplementation (with transitive = false) and adds an instrumented sanity test to validate retry behavior.
• Adds an Azure Pipelines PR validation YAML to boot an emulator and run Common’s instrumented tests, publishing logcat + HTML reports.

⚠️ Warning: Changes detected ⚠️

Please follow the recommendations in Adding or Updating Gradle Dependencies.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
testutils/src/main/java/com/microsoft/identity/internal/testutils/rules/FlakyTestRetryRule.kt	Adds an opt-in JUnit4 rule that retries annotated instrumented tests on-device.
testutils/src/main/java/com/microsoft/identity/internal/testutils/annotations/RetryFlakyTest.kt	Adds runtime annotation to configure retry count at method/class scope.
common/src/androidTest/java/com/microsoft/identity/common/RetryRuleValidationTest.kt	Adds an instrumented sanity test proving the rule can re-run and later pass.
common/build.gradle	Adds androidTestImplementation on :testutils (non-transitive) to consume the retry rule/annotation.
azure-pipelines/pull-request-validation/pr-common-instrumented.yml	Adds PR validation pipeline to run common:connectedLocalDebugAndroidTest on an emulator and publish artifacts.

[github-actions]

✅ Work item link check complete. Description contains link AB#3678341 to an Azure Boards work item.

[github-actions]

❌ Work item link check failed. Description contains AB#3678341 but the Bot could not link it to an Azure Boards work item.

Click here to learn more.

[shahzaibj]

transitive = false doesn't keep only the rule's classes though. it still puts ALL of testutils on the androidTest classpath (the keyvault/labapi/lab helpers, everything), it just drops testutils' transitive deps (gson, nimbus, robolectric, etc). so the whole graph is still there, just now half of it can't be used - any common instrumented test that touches those other testutils classes would fail to compile / NoClassDefFoundError since their deps are gone.

if the intent is just to keep robolectric and the JVM-only stuff off the instrumented classpath then fine, but the comment isn't accurate. do these 2 classes even need to live in testutils, or could they go somewhere lighter that common's androidTest can depend on cleanly?

[shahzaibj]

IIUC this re-runs base on the same test instance, so @Before/@after re-run each attempt but any field initialized at declaration (not in @before) carries over between attempts. The validation tests lean on that with their counters.

The KDoc pitches this as equivalent to the gradle test-retry plugin, but that one re-runs with a fresh instance I believe. So a stateful flaky test that mutates a field before it fails won't get a clean slate on retry here.

can we note that in the annotation KDoc so people writing flaky tests don't get surprised?

[shahzaibj]

when a test passes on retry we just Log.i to logcat and return, so in the test report a test that flaked twice then passed on the 3rd attempt looks identical to one that passed first try. doesn't that defeat the "stays opt-in so flaky tests remain visible" goal? the flakiness ends up buried in logcat and the result is a clean green.

is something downstream parsing these logs to surface the tests that are still flaking?

[shahzaibj]

Bit confused on the no-AB# path. Abort if no AB ID found (line 37) does exit 1 when abid is empty, which fails the job. But this branch treats an empty abid as skipped by design and exits 0. By the time we get here the abort already failed the run, so this exit 0 can't rescue it. So is a missing AB# meant to fail the PR or pass? The two steps disagree.