Skip to content

[tests] Disable known-flaky tests - #13646

Closed
simonrozsival wants to merge 4 commits into
mainfrom
simonrozsival-disable-known-flaky-tests
Closed

simonrozsival wants to merge 4 commits into
mainfrom
simonrozsival-disable-known-flaky-tests

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Context: #13429

These tests failed in isolation and then passed on retry of the same commit across many PRs between 2026-10-19 and 2026-11-18, leaving PR runs red. Disable each entire test until its flakiness is addressed. Each tracking issue records the evidence and criteria for re-enabling it. Keep DotNetNewAndroidTest and InstallAndroidDependenciesTest enabled because their coverage is important; the other nine tests remain disabled.

Test Tracking issue
PerformanceTest.Build_XAML_Change #13636
JavaMarshalGCBridgeTests.ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease #13637
JavaObjectTest.Dispose_Finalized #13638
TrimmableTypeMapTypeManagerTests.RegisteredPeer_Dispose_Finalized #13639
InstallAndRunTests.GradleFBProj #13640
InstallTests.IncrementalFastDeployment #13641
PingTest.PingLocalhost #13642
RuntimeEventSourceTests.GCBridgeRoundEmission #13645
JavaObjectTest.UnreferencedInstanceIsCollected #13275

Validation

  • git diff --check 5503f6a73f5d2a9a97f19d93a43761dbcacad6ca HEAD — passed.
  • dotnet-local.cmd build Microsoft.Android.slnx -nodeReuse:false — passed with 29 warnings.
  • dotnet-local.cmd test bin\TestDebug\net11.0\Xamarin.Android.Build.Tests.dll --list-tests --filter "Name~InstallAndroidDependenciesTest" — discovered both cases.
  • dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net11.0\MSBuildDeviceIntegration.dll --list-tests --filter "Name~DotNetNewAndroidTest" — discovered all four cases.
  • Tests were not executed because their Windows teardown stops the shared ADB server and devices used by other sessions.

Context: #13429

Disable 11 tests that failed and then passed on retry of the same
commit across many PRs between 2026-10-19 and 2026-11-18. Each test
is tracked by its own issue, which records the evidence and the
re-enable criteria.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 10:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the targeted quarantine of known-flaky tests.

No open issues 🎉

What changed in this PR

Temporarily disables 11 known-flaky tests across .NET for Android’s build and device test suites to reduce unrelated CI failures tracked in #13429. This quarantines the tests without fixing their underlying causes.

Changes:

  • Adds method-level [Ignore] attributes, covering all parameterized cases of each selected test.
  • Links every ignored test to its tracking issue while preserving test implementations.

Validation reported in the PR: git diff --check passed; builds and tests were not run.

File Description
tests/​MSBuildDeviceIntegration/​Tests/​PerformanceTest.cs Ignores the XAML-change performance test.
tests/​MSBuildDeviceIntegration/​Tests/​InstallTests.cs Ignores incremental fast-deployment tests.
tests/​MSBuildDeviceIntegration/​Tests/​InstallAndRunTests.cs Ignores Gradle integration and Android test-template tests.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​System.Net.NetworkInformation/​PingTest.cs Ignores the localhost ping test.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Java.Interop/​TrimmableTypeMapTypeManagerTests.cs Ignores registered-peer finalization testing.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Java.Interop/​JavaMarshalGCBridgeTests.cs Ignores the Java-rooted peer-cycle collection test.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Android.Runtime/​RuntimeEventSourceTests.cs Ignores GC bridge event-emission testing.
tests/​Mono.Android-Tests/​Java.Interop-Tests/​Java.Interop/​JavaObjectTest.cs Ignores two collection and finalization tests.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​AndroidDependenciesTests.cs Ignores Android dependency-installation tests.

Review effort: Balanced

@jonathanpeppers jonathanpeppers left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of the tests seem more important than the others.

[TestCase ("test", MSTestPackageChannel.Stable)]
[TestCase ("test", MSTestPackageChannel.Nightly)]
[TestCase ("list", MSTestPackageChannel.Stable)]
[Ignore ("Flaky: https://github.com/dotnet/android/issues/13643")]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one seems important to catch issues, though.

{
[Test]
[NonParallelizable] // Do not run environment modifying tests in parallel.
[Ignore ("Flaky: https://github.com/dotnet/android/issues/13644")]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one as well.

@jonathanpeppers jonathanpeppers Nov 19, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one failed on two recent PRs, and it actually needs me to fix a manifest URL: https://aka.ms/AndroidManifestFeed/d18-0

main uses a copy in the repo, but servicing branches use the live URL.

@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Nov 19, 2026
@simonrozsival

Copy link
Copy Markdown
Member Author

@jonathanpeppers I suggest merging this as is an then address each individual problem as soon as possible to reinstante those tests. I think it's OK if we disable even some of the more imporant tests for a few days. On the other hand, I don't mind fixing some of the tests before we merge this PR.

Keep the Android test template and dependency installation tests enabled to retain their coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival

Copy link
Copy Markdown
Member Author

InstallAndroidDependenciesTest failure in build 1690697

This does not look like the test's earlier intermittent download failures. The same failure occurred in five jobs across Linux, macOS, and Windows, on both CoreCLR and NativeAOT cases: the SDK install command returned success, but platform-tools@37.0.1 was not installed.

The build logs show the exact Android CLI command requesting platform-tools@37.0.1, followed by the installer warning that the package is missing. The build used Android CLI 1.1.16717390. Its stable repository currently lists platform-tools at 37.0.3; it does not offer 37.0.1. I reproduced the CI behavior with that exact CLI version:

$ android-cli --sdk=<empty-sdk> sdk install platform-tools@37.0.1
No url for platform-tools. Ignoring.
exit code: 0

The task then detects the missing package and emits a warning. Since the install target itself returns success, the test's retry loop does not retry; the test fails afterward on its explicit missing-dependency check. This is a stale package pin/catalog mismatch, not a transient network failure. XAPlatformToolsVersion is pinned to 37.0.1; it needs to be updated to a version available in the Android CLI repository, along with the corresponding per-OS archive pins/hashes.

Evidence: build 1690697, issue #13644, Android CLI install syntax, and Android CLI package-list syntax.

simonrozsival and others added 2 commits November 19, 2026 23:35
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival

Copy link
Copy Markdown
Member Author

Closing this blanket-disable PR in favor of targeted changes: #13676 removes Build_XAML_Change and IncrementalFastDeployment; #13675 addresses the GC bridge tests; #13666 addresses platform-tools version resolution. GradleFBProj (#13640) and PingLocalhost (#13642) remain enabled and are the two remaining investigations. No tracking issues are being closed by this action.

jonathanpeppers pushed a commit that referenced this pull request Nov 20, 2026
Fixes: #13637
Fixes: #13645
Fixes: #13638
Fixes: #13639
Fixes: #13275
Context: #13646

Several GC bridge tests fail intermittently on CI:

```
ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease
  Expected collection to not be empty. The Java-rooted cycle must participate in global-to-weak bridge processing.
GCBridgeRoundEmission
  GC bridge Start and Stop events should be emitted as matched pairs.
Dispose_Finalized / RegisteredPeer_Dispose_Finalized
  Expected ...Dispose(disposing: false) to run.
UnreferencedInstanceIsCollected
  Expected the unreferenced instance to be collected.
```

### Cause

The bridge itself works as designed; the tests' waits are wrong. Bridge rounds run one at a time on a background thread. While a round is running, a GC does not start another one, and `JNIEnv.BridgeProcessingGeneration` is incremented when a round ends. Each test made one of these assumptions:

  * **"The generation changed, so my peers were processed."** A round that started before the test's peers existed can finish during the wait (`ManagedPeerCycle...`, `JavaObjectArrayRetainsManagedValueUntilReleased`).
  * **"The next round writes Start/Stop events."** `ProcessBridge` checks whether the events are enabled once, when the round starts. A round already running when the listener is created writes neither event, but still bumps the generation, after which the test stops forcing GCs and waits 5 s for events that never come (`GCBridgeRoundEmission`).
  * **"The object will be finalized within 2 s."** A slow round, or one left over from an earlier test, uses up the whole budget (`Dispose_Finalized`, `RegisteredPeer_Dispose_Finalized`, `UnreferencedInstanceIsCollected`).

### Fix

`GCBridgeTestHelpers.WaitForBridgeRoundsAsync()` reads the generation and waits for three more rounds to complete, forcing a GC with a throwaway peer every 10 ms so a new round can always start. When the wait begins, one round may still be running and the next may already be queued with objects gathered by an earlier GC, so the third round to complete must have started from a GC after the wait began. That round has processed the test's peers. The second round's callback also started after the wait began, so its Start/Stop events have been written. `CollectUnreachablePeersAsync()` follows the wait with a GC, `WaitForPendingFinalizers()`, and a second GC.

The tests now assert directly after the wait instead of polling a predicate. Their deadline is `[Timeout (60_000, CooperativeCancellation = true)]`, which only fires if something is actually broken. `GCBridgeRoundEmission` now matches Start and Stop by event ID, because TypeMap events from other threads can be written between them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants