RELOPS-2433 + RELOPS-2458: Defender real-time hardening (hw fleet) + per-task RTP toggle (testers)

[markcor]

Summary

Hardens Windows Defender real-time / on-access scanning disable on the NUC13
hardware fleet (releng-hardware/win11-64-24h2-hw*), closing the WdFilter
boot race where a Defender platform update can re-arm on-access scanning
mid-life and inject scan overhead into Speedometer 3 / Talos / Raptor runs.

Tracked by RELOPS-2433.

Split note: this work was originally scrambled together with the fleetbench
PSU-degradation monitoring in #1235. That PR has since been split in two:

• RELOPS-2402: fleetbench hardware-health / PSU-degradation monitoring (Windows hw) #1235 → fleetbench hardware-health / PSU-degradation monitoring
  (RELOPS-2402).
• this PR (RELOPS-2433 + RELOPS-2458: Defender real-time hardening (hw fleet) + per-task RTP toggle (testers) #1264) → the Windows Defender hardening
  (RELOPS-2433).

Parent effort: RELOPS-2396
(perf-debug node benchmarking), where Defender on-access scanning was identified
as a contributor to bimodal SP3 scores.

Hardware-only: gated to the datacenter maintain-system path.

Findings that shaped this

• Get-MpComputerStatus / Set-MpPreference are unusable on the hw fleet
  ("Provider load failure") because the disable mechanism renames the Defender driver
  .sys files. Cmdlets can't manage what they can't load.
• Tamper Protection is enforced and not toggleable in-OS (TamperProtectionSource=5).
  fltmc unload WdFilter and sc config WdFilter start=disabled are both blocked under Tamper.
• WdFilter.sys is a boot-start minifilter: it loads at boot before the disable
  schtask renames it, so a Defender platform update that restores WdFilter.sys leaves
  real-time scanning active until the next boot (intermittently — observed live: nuc13-119
  had 38 WdFilter instances attached despite the .sys being renamed, nuc13-059 clean).

Changes

• profiles/disable_services.pp — datacenter case explicitly includes
  disable_windows_defender_schtask (de-coupled from the ReleaseId in ['2004','2009']
  coincidence that fired on 24H2 by accident) plus disable_windows_defender.
• win_disable_services::disable_windows_defender — rewritten registry-only:
  DisableAntiSpyware, the Real-Time Protection family, ForceDefenderPassiveMode,
  Sense\Start=4 (EDR off), and an idempotent exec disabling Defender's own scheduled tasks.
• maintainsystem-hw.ps1 → Invoke-DefenderRealtimeGuard — runs after
  Test-ConnectionUntilOnline: asserts GP path exclusions, re-renames restored
  WdFilter/WdBoot/WdNisDrv, tries fltmc unload, and reboots ONCE (boot-loop guarded:
  marker file, MaxReboots=1, CooldownMin=60) if WdFilter still runs under Tamper.
  Writes defender_status.json. Never throws.
• win_nsclient::check_defender — NEW external check + defender alias, reading
  ground-truth signals (WdFilter not loaded) rather than Get-MpComputerStatus.

Files

New: modules/win_nsclient/files/check_defender.ps1

Modified: disable_services.pp, disable_windows_defender.pp, win_nsclient/manifests/init.pp,
win_nsclient/templates/nsclient.ini.epp, win_scheduled_tasks/files/maintainsystem-hw.ps1
(Write-DefenderStatus + Invoke-DefenderRealtimeGuard + the guard call).

RELOPS-2458: Windows Defender real-time protection switch (driver-rename based)

Adds an on/off control for Defender real-time / on-access scanning on the CI hw fleet, in
the new win_defender_rt_toggle module. Default is OFF and non-persistent — enabling is
a transient, in-session testing override; every boot resets to disabled.

Mechanism: Set-MpPreference is unusable on this fleet (management platform absent →
"Provider load failure"; Tamper enforced), so the switch renames the boot-start minifilter
drivers (WdFilter/WdBoot/WdNisDrv) via takeown, below Tamper's reach.

• defender_rt_disable — user-triggerable (GRGX): rename drivers (off; effective next boot).
• defender_rt_enable — user-triggerable (GRGX): transient restore + best-effort fltmc load so scanning runs this session for a test. Not persisted.
• defender_rt_enable_boot — boot reconcile that always enforces OFF (ignores any prior enable), catches a Defender update that restored the driver, and does a boot-loop-guarded single reboot to clear a still-running filter.

Triggering the switch (task payload, or manual) — the tasks are ACL'd GRGX, so the unprivileged task user can run but not modify them; they execute as SYSTEM and return immediately:

# Enable Defender real-time protection for a test (transient - a reboot resets to OFF)
schtasks.exe /Run /TN "defender_rt_enable"

# Disable / restore the default (off)
schtasks.exe /Run /TN "defender_rt_disable"

Cmdlet equivalent: Start-ScheduledTask -TaskName defender_rt_enable. State: C:\ProgramData\defender_rt\rt_state.json (and defender_status.json).

Security model: GRGX trigger-only for BUILTIN\Users (no FA), boot task untouchable,
idempotent -Check registrar. Consolidated to single authority: disable_services.pp
(datacenter) includes this instead of disable_windows_defender_schtask;
Invoke-DefenderRealtimeGuard removed from maintainsystem-hw.ps1.

Validated on nuc13-059: off-by-default after re-image; unprivileged schtasks /Run of
defender_rt_enable flips RTP on (runs as SYSTEM, LastTaskResult=0); after reboot the
reconcile enforced OFF (ignored the on-marker).

⚠️ Known limitation (proven on nuc13-059). The switch works for the normal case — default OFF, transient enable, reboot resets to OFF — as long as the Defender platform has not been populated. If Defender pulls a platform update while enabled, it stages the active driver at system32\drivers\wd\WdFilter.sys (ImagePath repointed) and the rename-based disable can no longer hold: the authoritative driver copies under C:\ProgramData\Microsoft\Windows Defender\Platform\<ver>\Drivers\ are Tamper-protected (rename = Access denied), and WinDefend re-stages the driver from there on every boot. Verified: after renaming the inbox + wd\ copies and rebooting, WdFilter came back RUNNING, re-staged from the Tamper-protected Platform copy; Set-MpPreference is likewise reverted by Tamper (IsTamperProtected=True). So a platform-updated node cannot be returned to disabled in-OS — it needs a re-image (back to empty-platform) or Tamper Protection OFF in the image (RELOPS-2433 §4f), after which Set-MpPreference is a clean reversible switch. Mitigation: keep enable (test) windows short so a platform update is less likely to land.

Testing

Testing on the win11-64-24h2-hw-perf-debug pool. Try push:
https://treeherder.mozilla.org/jobs?repo=try&landoInstance=lando-prod-2025&landoCommitID=66499

End-to-end validated on nuc13-059 (6a21436): defender_rt_enable → repuppetize=1 → defender-only puppet re-apply (exit 2, ~4.5s, tasks re-registered) → reboot resets to OFF. Meets the requirement (default off / transient enable / reboot resets to off).

Follow-on work

• Restore WdFilter\Start=4 in disable_windows_defender.pp so the kernel cannot load
  the minifilter at boot at all — makes the boot race structurally impossible on
  Tamper-off nodes (per RELOPS-2433 root-cause analysis).
• Roll the hardening from -perf-debug + -alpha to the remaining win11-64-24h2-hw* pools.
• RELOPS-2434 — post-deploy
  Pester sampling on real hw nodes to catch the next regression of this shape.

🤖 Generated with Claude Code