Skip to content

dev-guide.md

Latest commit

 

History

History
245 lines (169 loc) · 14.5 KB

dev-guide.md

File metadata and controls

245 lines (169 loc) · 14.5 KB
Raw

Development Guide

This document contains all the required information to build, test, and consume MSTest.

Prerequisites

To build and test all functionalities of MSTest, we recommend installing Visual Studio 2026 with the following workloads:

  • .NET desktop development
  • Universal Windows Platform development
  • .NET Core cross-platform development

Recommended workflow

We recommend the following overall workflow when developing this repository:

  • Fork this repository.
  • Always work on your fork.
  • Always keep your fork up to date.

Starting with your fork

Before updating your fork, run this command:

git remote add upstream https://github.com/Microsoft/testfx.git

This will make management of multiple forks and your work easier over time.

Updating your fork

We recommend the following commands to update your fork:

git checkout main
git clean -dfx
git fetch upstream
git rebase upstream/main
git push

Or more succinctly:

git checkout main && git clean -xdf && git fetch upstream && git rebase upstream/main && git push

This will update your fork with the latest from microsoft/testfx on your machine and push those updates to your remote fork.

Building

The easiest and recommended solution is to build the repository with the provided scripts at the repo root.

For Windows:

build.cmd

For Linux and macOS:

./build.sh

Common building options

By default, the script generates a Debug build type, which is not optimized code and includes asserts. As its name suggests, this makes it easier and friendlier to debug the code. If you want to make performance measurements, you ought to build the Release version instead, which doesn't have any asserts and has all code optimizations enabled. Likewise, if you plan on running tests, the Release configuration is more suitable since it's considerably faster than the Debug one. For this, you add the flag -configuration release (or -c release). For example:

For Windows:

build.cmd -configuration release

For Linux and macOS:

./build.sh --configuration release

Another common flag is -pack which will produce the NuGet packages of MSTest. These packages are required for the acceptance tests (see testing section).

For more information about all the different options available, supply the argument -help|-h when invoking the build script. On Unix-like systems, non-abbreviated arguments can be passed in with a single - or double hyphen --.

MSBuildCache

The Windows PR pipeline experimentally runs MSBuildCache. The cache-aware build uses Arcade's eng/common/msbuild.ps1 launcher to invoke the solution directly because Arcade's outer Build.proj discovers projects dynamically and cannot expose the repository's static project graph to the cache plugin. PR builds consume the immutable Azure Pipeline cache read-only, and fork PRs skip this step because they do not receive the required token scope.

The cache steps have three outcomes:

  • The cache build succeeds. The pipeline runs the remaining Arcade restore, sign, and pack phases without rebuilding, then uses the cached outputs for the test steps. The regular Arcade Build step is skipped.
  • The cache build fails with errors the regular build would only reproduce. A cache hit materializes previously stored outputs and never runs the compiler, so an ordinary build error came from a project that genuinely executed, exactly as the fallback would execute it. Re-running the whole solution through Arcade would report the same errors several minutes later, so the pipeline surfaces them on the cache step and fails the job there instead. Should such a verdict ever be wrong, the cache diagnostics published under artifacts\log\<configuration>\MSBuildCache show what the plugin did for that run.
  • The cache build or the preparation phase fails for any other reason. The pipeline preserves the cache diagnostics, removes partial outputs, and runs the regular Arcade build as a fallback. This covers anything MSBuildCache itself reports (plugin, cache, or file-access problems, including its duplicate-output diagnostic), NU* restore failures, engine crashes and OOM (MSB4166, MSB0001, MSB1025, MSB4017), MSB4260 project references that cannot be resolved with a static graph (a /graph limitation the fallback does not have), transient file locks (MSB3021, MSB3027), an unexpected failure of the wrapper script itself, and any failure that never reached MSBuild's error summary. The cache step stays green in this case: marking it SucceededWithIssues would make the job PartiallySucceeded, which Azure Repos build-validation policies treat as a failure even when the fallback build then succeeds.

Every merge to main that touches product build inputs runs a dedicated, batched seed stage for both Debug and Release. This is required: the cache fingerprints project inputs, so entries become stale whenever shared build inputs such as global.json, eng/Versions.props, or Arcade change. This stage is the only remote cache publisher; PR, manual, and nightly canary builds consume the cache read-only so they cannot race to publish immutable entries. Debug and Release use separate cache universes because configuration-independent projects can otherwise race while the two configurations publish in parallel.

The cache's detached-process exclusions use fully rooted paths (for example, $(WinDir)\**). A drive-relative pattern such as \Windows\** does not match the absolute file-access paths reported by MSBuild and causes otherwise successful cache builds to fail after compilation. Cache builds also pass -warnAsError:$false to Arcade's launcher because MSBuildCache intentionally warns about allowlisted detached telemetry accesses. The fallback Arcade build still treats warnings as errors.

Two settings exist purely to keep the cache usable in this pipeline, and both must stay in sync between the seed stage and the PR canary:

  • NUGET_PACKAGES is pinned to $(Build.SourcesDirectory)\.packages\ on both steps. MSBuildCache builds its path normalizer from NUGET_PACKAGES and fingerprints every restored package file as a project input, so restoring to a different folder changes every node's fingerprint and misses the entire graph. The seed stage gets that folder implicitly from Arcade's -ci switch; the canary cannot pass -ci, because eng/common/tools.ps1 would then report build failures through Write-PipelineSetResult and exit with code 0, which would make the pipeline treat a failed cache build as a success. Pinning the variable explicitly keeps the two identical without relying on -ci.
  • MSBuildCacheIdenticalDuplicateOutputPatterns is set to \** in Directory.Build.props. By default MSBuildCache hardlinks every output into its content-addressable store and marks it read-only, both when materializing a cache hit and when ingesting the outputs of a cache miss. That makes the Arcade restore/sign/pack pass that follows the cached graph build fail with UnauthorizedAccessException while rewriting files such as artifacts\bin\**\*.deps.json or the generated src\Package\MSTest.Sdk\Sdk\Sdk.props. Clearing the read-only attribute is not a valid workaround: the output file and the cache entry share an inode, so writing through it corrupts the cache and can silently rewrite a same-content output owned by another project. Matching every output switches the plugin to copy semantics, which leaves the outputs writable. The trade-offs are extra I/O and a relaxed duplicate-output check that tolerates two projects writing the same path when the content is identical; writing the same path with different content is still an error.

Changing either setting invalidates the existing cache entries, so PR runs miss until the next main merge reseeds them.

MSBuildCache currently requires Windows, the Visual Studio version pinned in global.json, Git on PATH, and a clean repository. It does not support incremental developer builds. To validate the local cache from a clean checkout, run:

eng\common\msbuild.ps1 -msbuildEngine vs -warnAsError:$false TestFx.slnx /restore /graph /m /reportfileaccesses /t:Build /p:Configuration=Release /p:FastAcceptanceTest=true /p:Publish=false /p:Test=false /p:MSBuildCachePackageEnabled=true /p:MSBuildCacheEnabled=true /p:MSBuildCacheLogDirectory=artifacts\log\Release\MSBuildCache\Plugin /bl:artifacts\log\Release\MSBuildCache\Build.binlog

Use MSTest.slnf or Microsoft.Testing.Platform.slnf in place of TestFx.slnx to validate a filtered graph. Delete artifacts\msbuild-cache to clear the local content cache. Change the prefix of MSBuildCacheCacheUniverse in Directory.Build.props, or pass /p:MSBuildCacheCacheUniverse=<new-value> for one invocation, to invalidate cache entries. The default universe includes $(Configuration), so use the same configuration when populating and consuming a cache.

Build layout

MSTest uses Microsoft common infrastructure called arcade as such all outputs follow this structure:

artifacts
  bin
    $(MSBuildProjectName)
      $(Configuration)
  packages
    $(Configuration)
      Shipping
        $(MSBuildProjectName).$(PackageVersion).nupkg
      NonShipping
        $(MSBuildProjectName).$(PackageVersion).nupkg
      Release
      PreRelease
  TestResults
    $(Configuration)
      $(MSBuildProjectName)_$(TargetFramework)_$(TestArchitecture).(xml|html|log|error.log)
  SymStore
    $(Configuration)
      $(MSBuildProjectName)
  log
    $(Configuration)
      Build.binlog
  tmp
    $(Configuration)
  obj
    $(MSBuildProjectName)
      $(Configuration)
  toolset

with

directory description
bin Build output of each project.
obj Intermediate directory for each project.
packages NuGet packages produced by all projects in the repo.
TestResults Test results produced by test runs.
SymStore Storage for converted Windows PDBs
log Build binary log and other logs.
tmp Temp files generated during build.
toolset Files generated during toolset restore.

Testing

MSTest uses the following 3 kinds of tests:

  • Unit tests
    • Very fast tests primarily validating individual units.
    • Named as <ProjectUnderTest>.UnitTests where <ProjectUnderTest> is the project under test.
  • Integration tests
    • Slightly slower tests with File system interactions.
    • Named either as <ProjectUnderTest>.IntegrationTests or as <PackageUnderTest>.Acceptance.IntegrationTests where <ProjectUnderTest> is the project under test <PackageUnderTest> is the package under test
  • Performance tests
    • Focused tests that ensure the performance of specific workflows of the application

The easiest way to run the tests is to call

For Windows:

build.cmd -pack -test -integrationTest

For Linux and macOS:

./build.sh -pack -test -integrationTest

Note that -test allows to run the unit tests and -integrationTest allows to run the two kinds of integration tests. Acceptance integration tests require the NuGet packages to have been produced hence the -pack flag.

Working with Visual Studio

If you are working with Visual Studio, we recommend opening it through the open-vs.cmd script at the repo root. This script will set all the required environment variables required so that Visual Studio picks up the locally downloaded version of the .NET SDK. If you prefer to use your machine-wide configuration, you can open Visual Studio directly.

Inside Visual Studio, all projects can be built normally. All but acceptance tests can be tested directly from Visual Studio. The acceptance tests will always use the version of the NuGet packages produced in the artifacts/packages/shipping folder so if you have made some changes and run these tests, it's likely that the changes will not be applied.

Developing MSTest.Sdk

Do not use IsImplicitlyDefined="true" on PackageReference items in the MSTest.Sdk .targets files. The package would be defined twice, which can produce NU1009 warnings that are commonly treated as errors.

Do not use VersionOverride on those PackageReference items. Although it can override a version under Central Package Management (CPM), it is forbidden when CentralPackageVersionOverrideEnabled is false and causes NU1013.

Instead, split version specification based on CPM:

  • When ManagePackageVersionsCentrally is not true, set Version directly on the PackageReference.
  • When ManagePackageVersionsCentrally is true, leave the PackageReference unversioned and add a matching PackageVersion item.

This supports both values of CentralPackageVersionOverrideEnabled without producing NU1009.

Layering MSTest.Sdk on another base SDK

MSTest.Sdk implicitly imports Microsoft.NET.Sdk. To combine it with another base SDK, such as Microsoft.NET.Sdk.Web, import both SDKs manually and list the other SDK first:

<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk.Web" />
  <Import Project="Sdk.props" Sdk="MSTest.Sdk" />

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
  </PropertyGroup>

  <Import Project="Sdk.targets" Sdk="MSTest.Sdk" />
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk.Web" />
</Project>

Because an <Import> cannot specify the SDK version in the same way as <Project Sdk="MSTest.Sdk/x.y.z">, pin it in global.json:

{
  "msbuild-sdks": {
    "MSTest.Sdk": "x.y.z"
  }
}

Sdk.props and Sdk.targets guard their Microsoft.NET.Sdk imports with _MSTestSdkImportsMicrosoftNETSdk. MSTest.Sdk imports the base SDK only when another SDK has not already set UsingMicrosoftNETSdk, avoiding MSB4011 duplicate-import warnings.

Visual Studio version requirement

If working with Visual Studio, this repository uses the new, modern, XML-based slnx solution file format (TestFx.slnx). This solution file can only be opened or loaded successfully using Visual Studio 2022 17.13 or higher. Opening the TestFx.slnx directly with a different version of Visual Studio installed other than Visual Studio 2022 17.13 or higher will just open the slnx file in a raw solution XML format.