-
Notifications
You must be signed in to change notification settings - Fork 0
Add LLVM coverage instrumentation workflow #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| --- | ||
| title: Coverage Guide | ||
| summary: How to measure LLVM source coverage for greenflame_core. | ||
| audience: contributors | ||
| status: authoritative | ||
| owners: | ||
| - core-team | ||
| last_updated: 2026-04-06 | ||
| tags: | ||
| - coverage | ||
| - llvm-cov | ||
| - clang | ||
| --- | ||
|
|
||
| # Coverage Guide | ||
|
|
||
| This document describes how to measure source coverage for `greenflame_core` | ||
| using LLVM's instrumentation-based coverage (`llvm-cov`). | ||
|
|
||
| Coverage is a diagnostic tool, not a gate. It helps identify untested paths in | ||
| `greenflame_core`. It is not required before considering a task complete. | ||
|
|
||
| --- | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| In addition to the standard build prerequisites in [build.md](build.md): | ||
|
|
||
| - **clang-cl.exe** — provided by the Visual Studio 2026 "C++ Clang compiler | ||
| for Windows" component (already required for the `x64-debug-clang` preset). | ||
| - **llvm-profdata.exe** and **llvm-cov.exe** — these ship with the VS | ||
| "C++ Clang compiler for Windows" component at: | ||
| ``` | ||
| <VS install>\VC\Tools\Llvm\x64\bin\ | ||
| ``` | ||
| `scripts\coverage.ps1` adds this directory to `PATH` automatically when | ||
| `VSINSTALLDIR` is set (i.e. when run from a VS Developer Command Prompt or | ||
| after running `VsDevCmd.bat`). In a plain shell, set `VSINSTALLDIR` or add | ||
| the directory to `PATH` manually. | ||
|
|
||
| --- | ||
|
|
||
| ## Running coverage | ||
|
|
||
| A helper script handles the full workflow: | ||
|
|
||
| ```powershell | ||
| .\scripts\coverage.ps1 | ||
| ``` | ||
|
|
||
| The script: | ||
| 1. Configures with the `x64-coverage-clang` CMake preset (Clang + coverage | ||
| instrumentation enabled). | ||
| 2. Builds `greenflame_tests`. | ||
| 3. Runs the test binary with `LLVM_PROFILE_FILE` set to capture raw profile | ||
| data. | ||
| 4. Merges the raw profile with `llvm-profdata`. | ||
| 5. Prints a summary coverage table scoped to `src\greenflame_core\`. | ||
| 6. Writes an HTML line-level report to | ||
| `build\x64-coverage-clang\coverage\report\index.html`. | ||
|
|
||
| To open the HTML report after the run: | ||
|
|
||
| ```powershell | ||
| start build\x64-coverage-clang\coverage\report\index.html | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## How it works | ||
|
|
||
| The `x64-coverage-clang` CMake preset sets `GREENFLAME_ENABLE_COVERAGE=ON`. | ||
| When enabled, CMake adds the following flags to `greenflame_core` and | ||
| `greenflame_tests`: | ||
|
|
||
| | Target | Compile flags | Link flags | | ||
| |---------------------|------------------------------------------------------|-------------------------------| | ||
| | `greenflame_core` | `-fprofile-instr-generate -fcoverage-mapping` | — | | ||
| | `greenflame_tests` | `-fprofile-instr-generate -fcoverage-mapping` | `-fprofile-instr-generate` | | ||
|
|
||
| Because `greenflame_core` is a static library, its instrumented object files | ||
| are linked into `greenflame_tests.exe`, which is the binary passed to | ||
| `llvm-cov`. | ||
|
|
||
| --- | ||
|
|
||
| ## Manual workflow | ||
|
|
||
| If you prefer to run the steps yourself: | ||
|
|
||
| ```powershell | ||
| # 1. Configure and build | ||
| cmake --preset x64-coverage-clang | ||
| cmake --build --preset x64-coverage-clang | ||
|
|
||
| # 2. Run tests (generates coverage.profraw) | ||
| $env:LLVM_PROFILE_FILE = "$PWD\build\x64-coverage-clang\coverage\coverage.profraw" | ||
| .\build\x64-coverage-clang\bin\greenflame_tests.exe | ||
| Remove-Item Env:\LLVM_PROFILE_FILE | ||
|
|
||
| # 3. Merge profile data | ||
| llvm-profdata merge -sparse ` | ||
| build\x64-coverage-clang\coverage\coverage.profraw ` | ||
| -o build\x64-coverage-clang\coverage\coverage.profdata | ||
|
|
||
| # 4. Summary report (scoped to greenflame_core) | ||
| llvm-cov report ` | ||
| .\build\x64-coverage-clang\bin\greenflame_tests.exe ` | ||
| "--instr-profile=build\x64-coverage-clang\coverage\coverage.profdata" ` | ||
| src\greenflame_core | ||
|
|
||
| # 5. HTML report | ||
| llvm-cov show ` | ||
| .\build\x64-coverage-clang\bin\greenflame_tests.exe ` | ||
| "--instr-profile=build\x64-coverage-clang\coverage\coverage.profdata" ` | ||
| "--output-dir=build\x64-coverage-clang\coverage\report" ` | ||
| -format=html ` | ||
| src\greenflame_core | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Scope | ||
|
|
||
| The report is scoped to `src\greenflame_core\`. GoogleTest internals and test | ||
| source files under `tests\` are excluded from the summary by passing the core | ||
| source directory as a positional filter to `llvm-cov`. | ||
|
|
||
| Coverage of `src\greenflame\` (the Win32 GUI layer) is intentionally excluded: | ||
| that code cannot run in the headless test binary and is not a target for | ||
| automated coverage. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| <# | ||
| .SYNOPSIS | ||
| Build greenflame_core with LLVM coverage instrumentation, run tests, and | ||
| produce a coverage report scoped to src/greenflame_core/. | ||
|
|
||
| .DESCRIPTION | ||
| Requires: | ||
| - clang-cl.exe, llvm-profdata.exe, and llvm-cov.exe | ||
| These ship with the Visual Studio "C++ Clang compiler for Windows" | ||
| component at: | ||
| <VS install>\VC\Tools\Llvm\x64\bin\ | ||
| The script adds that directory to PATH automatically when $env:VSINSTALLDIR | ||
| is set (i.e. when run from a VS Developer Command Prompt or after | ||
| invoking VsDevCmd.bat). If you run from a plain shell, either set | ||
| VSINSTALLDIR or add the Llvm\x64\bin directory to PATH manually. | ||
|
|
||
| Produces (all under build\x64-coverage-clang\coverage\): | ||
| - coverage.profraw raw instrumentation profile | ||
| - coverage.profdata merged profile | ||
| - report\ HTML line-level coverage report | ||
| A summary table scoped to src/greenflame_core/ is printed to stdout. | ||
|
|
||
| .EXAMPLE | ||
| # From the repo root: | ||
| .\scripts\coverage.ps1 | ||
|
|
||
| # Open the HTML report after the script finishes: | ||
| start build\x64-coverage-clang\coverage\report\index.html | ||
| #> | ||
|
|
||
| $ErrorActionPreference = "Stop" | ||
|
|
||
| # ── resolve repo root ────────────────────────────────────────────────────────── | ||
| $RepoRoot = Split-Path -Parent $PSScriptRoot | ||
| Set-Location $RepoRoot | ||
|
|
||
| # ── add VS LLVM bin to PATH if needed ───────────────────────────────────────── | ||
| # llvm-cov and llvm-profdata ship with the VS "C++ Clang" component at: | ||
| # <VS install>\VC\Tools\Llvm\x64\bin\ | ||
| # Prepend that directory when $env:VSINSTALLDIR is set and the tools aren't | ||
| # already on PATH. | ||
| if ($env:VSINSTALLDIR -and | ||
| -not (Get-Command "llvm-cov" -ErrorAction SilentlyContinue)) { | ||
| $vsLlvmBin = Join-Path $env:VSINSTALLDIR "VC\Tools\Llvm\x64\bin" | ||
| if (Test-Path $vsLlvmBin) { | ||
| $env:PATH = "$vsLlvmBin;$env:PATH" | ||
| } | ||
| } | ||
|
|
||
| # ── verify required tools ────────────────────────────────────────────────────── | ||
| $missing = @() | ||
| foreach ($tool in @("cmake", "ninja", "llvm-profdata", "llvm-cov")) { | ||
| if (-not (Get-Command $tool -ErrorAction SilentlyContinue)) { | ||
| $missing += $tool | ||
| } | ||
| } | ||
| if ($missing.Count -gt 0) { | ||
| Write-Error @" | ||
| The following tools were not found on PATH: $($missing -join ', ') | ||
| See docs/coverage.md for setup instructions. | ||
| "@ | ||
| } | ||
|
|
||
| # ── configure ───────────────────────────────────────────────────────────────── | ||
| Write-Host "=== Configure (x64-coverage-clang) ===" | ||
| cmake --preset x64-coverage-clang | ||
| if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } | ||
|
|
||
| # ── build ───────────────────────────────────────────────────────────────────── | ||
| Write-Host "" | ||
| Write-Host "=== Build (x64-coverage-clang) ===" | ||
| cmake --build --preset x64-coverage-clang | ||
| if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } | ||
|
|
||
| # ── paths ───────────────────────────────────────────────────────────────────── | ||
| $TestExe = "build\x64-coverage-clang\bin\greenflame_tests.exe" | ||
| $CovDir = "build\x64-coverage-clang\coverage" | ||
| $ProfRaw = "$CovDir\coverage.profraw" | ||
| $ProfData = "$CovDir\coverage.profdata" | ||
| $HtmlDir = "$CovDir\report" | ||
| $CoreSrc = "src\greenflame_core" | ||
|
|
||
| New-Item -ItemType Directory -Force -Path $CovDir | Out-Null | ||
|
|
||
| # ── run tests with LLVM instrumentation ─────────────────────────────────────── | ||
| Write-Host "" | ||
| Write-Host "=== Run tests ===" | ||
| $env:LLVM_PROFILE_FILE = (Resolve-Path -LiteralPath $CovDir).Path + "\coverage.profraw" | ||
| & $TestExe | ||
| $TestExitCode = $LASTEXITCODE | ||
| Remove-Item Env:\LLVM_PROFILE_FILE -ErrorAction SilentlyContinue | ||
|
|
||
| if ($TestExitCode -ne 0) { | ||
| Write-Warning "Tests reported failures (exit $TestExitCode). Coverage data was still captured." | ||
| } | ||
|
|
||
| # ── merge profile data ──────────────────────────────────────────────────────── | ||
| Write-Host "" | ||
| Write-Host "=== Merge profile data ===" | ||
| llvm-profdata merge -sparse $ProfRaw -o $ProfData | ||
|
Comment on lines
+97
to
+100
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The merge step unconditionally consumes Useful? React with 👍 / 👎. |
||
| if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } | ||
|
|
||
| # ── summary report (stdout, scoped to greenflame_core) ──────────────────────── | ||
| Write-Host "" | ||
| Write-Host "=== Coverage summary (src\greenflame_core) ===" | ||
| llvm-cov report $TestExe "--instr-profile=$ProfData" $CoreSrc | ||
| if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } | ||
|
|
||
| # ── HTML report ─────────────────────────────────────────────────────────────── | ||
| Write-Host "" | ||
| Write-Host "=== HTML report -> $HtmlDir ===" | ||
| llvm-cov show $TestExe ` | ||
| "--instr-profile=$ProfData" ` | ||
| "--output-dir=$HtmlDir" ` | ||
| -format=html ` | ||
| $CoreSrc | ||
| if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } | ||
|
|
||
| Write-Host "" | ||
| Write-Host "Done. Open: $HtmlDir\index.html" | ||
| exit $TestExitCode | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PATH bootstrap only runs when
llvm-covis absent, but the script later requires bothllvm-covandllvm-profdata. On machines wherellvm-covis already on PATH (for example from another LLVM install) butllvm-profdatais missing or version-mismatched, this block skips adding Visual Studio's LLVM bin and the run fails or mixes incompatible tool versions. Gate this on both tools (or always prepend VS LLVM whenVSINSTALLDIRis set) so the toolchain is consistent.Useful? React with 👍 / 👎.