Repository navigation
[tests] Disable known-flaky tests - #13646
simonrozsival wants to merge 4 commits into
Conversation
Context: #13429 Disable 11 tests that failed and then passed on retry of the same commit across many PRs between 2026-10-19 and 2026-11-18. Each test is tracked by its own issue, which records the evidence and the re-enable criteria. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified for the targeted quarantine of known-flaky tests.
No open issues 🎉
What changed in this PR
Temporarily disables 11 known-flaky tests across .NET for Android’s build and device test suites to reduce unrelated CI failures tracked in #13429. This quarantines the tests without fixing their underlying causes.
Changes:
- Adds method-level
[Ignore]attributes, covering all parameterized cases of each selected test. - Links every ignored test to its tracking issue while preserving test implementations.
Validation reported in the PR: git diff --check passed; builds and tests were not run.
| File | Description |
|---|---|
| tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs | Ignores the XAML-change performance test. |
| tests/MSBuildDeviceIntegration/Tests/InstallTests.cs | Ignores incremental fast-deployment tests. |
| tests/MSBuildDeviceIntegration/Tests/InstallAndRunTests.cs | Ignores Gradle integration and Android test-template tests. |
| tests/Mono.Android-Tests/Mono.Android-Tests/System.Net.NetworkInformation/PingTest.cs | Ignores the localhost ping test. |
| tests/Mono.Android-Tests/Mono.Android-Tests/Java.Interop/TrimmableTypeMapTypeManagerTests.cs | Ignores registered-peer finalization testing. |
| tests/Mono.Android-Tests/Mono.Android-Tests/Java.Interop/JavaMarshalGCBridgeTests.cs | Ignores the Java-rooted peer-cycle collection test. |
| tests/Mono.Android-Tests/Mono.Android-Tests/Android.Runtime/RuntimeEventSourceTests.cs | Ignores GC bridge event-emission testing. |
| tests/Mono.Android-Tests/Java.Interop-Tests/Java.Interop/JavaObjectTest.cs | Ignores two collection and finalization tests. |
| src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidDependenciesTests.cs | Ignores Android dependency-installation tests. |
Review effort: Balanced
jonathanpeppers
left a comment
There was a problem hiding this comment.
Two of the tests seem more important than the others.
| [TestCase ("test", MSTestPackageChannel.Stable)] | ||
| [TestCase ("test", MSTestPackageChannel.Nightly)] | ||
| [TestCase ("list", MSTestPackageChannel.Stable)] | ||
| [Ignore ("Flaky: https://github.com/dotnet/android/issues/13643")] |
There was a problem hiding this comment.
This one seems important to catch issues, though.
| { | ||
| [Test] | ||
| [NonParallelizable] // Do not run environment modifying tests in parallel. | ||
| [Ignore ("Flaky: https://github.com/dotnet/android/issues/13644")] |
There was a problem hiding this comment.
This one failed on two recent PRs, and it actually needs me to fix a manifest URL: https://aka.ms/AndroidManifestFeed/d18-0
main uses a copy in the repo, but servicing branches use the live URL.
|
@jonathanpeppers I suggest merging this as is an then address each individual problem as soon as possible to reinstante those tests. I think it's OK if we disable even some of the more imporant tests for a few days. On the other hand, I don't mind fixing some of the tests before we merge this PR. |
Keep the Android test template and dependency installation tests enabled to retain their coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This reverts commit f46494b.
|
Closing this blanket-disable PR in favor of targeted changes: #13676 removes Build_XAML_Change and IncrementalFastDeployment; #13675 addresses the GC bridge tests; #13666 addresses platform-tools version resolution. GradleFBProj (#13640) and PingLocalhost (#13642) remain enabled and are the two remaining investigations. No tracking issues are being closed by this action. |
Fixes: #13637 Fixes: #13645 Fixes: #13638 Fixes: #13639 Fixes: #13275 Context: #13646 Several GC bridge tests fail intermittently on CI: ``` ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease Expected collection to not be empty. The Java-rooted cycle must participate in global-to-weak bridge processing. GCBridgeRoundEmission GC bridge Start and Stop events should be emitted as matched pairs. Dispose_Finalized / RegisteredPeer_Dispose_Finalized Expected ...Dispose(disposing: false) to run. UnreferencedInstanceIsCollected Expected the unreferenced instance to be collected. ``` ### Cause The bridge itself works as designed; the tests' waits are wrong. Bridge rounds run one at a time on a background thread. While a round is running, a GC does not start another one, and `JNIEnv.BridgeProcessingGeneration` is incremented when a round ends. Each test made one of these assumptions: * **"The generation changed, so my peers were processed."** A round that started before the test's peers existed can finish during the wait (`ManagedPeerCycle...`, `JavaObjectArrayRetainsManagedValueUntilReleased`). * **"The next round writes Start/Stop events."** `ProcessBridge` checks whether the events are enabled once, when the round starts. A round already running when the listener is created writes neither event, but still bumps the generation, after which the test stops forcing GCs and waits 5 s for events that never come (`GCBridgeRoundEmission`). * **"The object will be finalized within 2 s."** A slow round, or one left over from an earlier test, uses up the whole budget (`Dispose_Finalized`, `RegisteredPeer_Dispose_Finalized`, `UnreferencedInstanceIsCollected`). ### Fix `GCBridgeTestHelpers.WaitForBridgeRoundsAsync()` reads the generation and waits for three more rounds to complete, forcing a GC with a throwaway peer every 10 ms so a new round can always start. When the wait begins, one round may still be running and the next may already be queued with objects gathered by an earlier GC, so the third round to complete must have started from a GC after the wait began. That round has processed the test's peers. The second round's callback also started after the wait began, so its Start/Stop events have been written. `CollectUnreachablePeersAsync()` follows the wait with a GC, `WaitForPendingFinalizers()`, and a second GC. The tests now assert directly after the wait instead of polling a predicate. Their deadline is `[Timeout (60_000, CooperativeCancellation = true)]`, which only fires if something is actually broken. `GCBridgeRoundEmission` now matches Start and Stop by event ID, because TypeMap events from other threads can be written between them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Context: #13429
These tests failed in isolation and then passed on retry of the same commit across many PRs between 2026-10-19 and 2026-11-18, leaving PR runs red. Disable each entire test until its flakiness is addressed. Each tracking issue records the evidence and criteria for re-enabling it. Keep
DotNetNewAndroidTestandInstallAndroidDependenciesTestenabled because their coverage is important; the other nine tests remain disabled.PerformanceTest.Build_XAML_ChangeJavaMarshalGCBridgeTests.ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterReleaseJavaObjectTest.Dispose_FinalizedTrimmableTypeMapTypeManagerTests.RegisteredPeer_Dispose_FinalizedInstallAndRunTests.GradleFBProjInstallTests.IncrementalFastDeploymentPingTest.PingLocalhostRuntimeEventSourceTests.GCBridgeRoundEmissionJavaObjectTest.UnreferencedInstanceIsCollectedValidation
git diff --check 5503f6a73f5d2a9a97f19d93a43761dbcacad6ca HEAD— passed.dotnet-local.cmd build Microsoft.Android.slnx -nodeReuse:false— passed with 29 warnings.dotnet-local.cmd test bin\TestDebug\net11.0\Xamarin.Android.Build.Tests.dll --list-tests --filter "Name~InstallAndroidDependenciesTest"— discovered both cases.dotnet-local.cmd test bin\TestDebug\MSBuildDeviceIntegration\net11.0\MSBuildDeviceIntegration.dll --list-tests --filter "Name~DotNetNewAndroidTest"— discovered all four cases.