Conversation
Split out of PR #1235 (combined RELOPS-2396 branch) into fleetbench-only. Per NUC13 hardware node, before worker-runner starts: - win_fleetbench module installs the version-pinned collector to C:\fleetbench plus per-hardware baselines, wired via the hardware_observability profile. - maintainsystem-hw.ps1 runs `fleetbench cpu --mode quick --duration 900s --json` once post-bootstrap then at most once per 72h; 900s self-warms the node so PSU/thermal throttling surfaces (a short cold-boot run can false-pass). - Evaluates GOOD/BAD/MARGINAL/UNKNOWN vs the locked per-hardware baseline, plus drift vs the node's first recorded run. - NSClient++ checks `fleetbench` and `fleetbench_variance` surface verdict + metrics to Marlin (Icinga2/Grafana). Hardware-only: gated to the datacenter maintain-system path + hardware_observability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Invoke-FleetbenchCheck runs `fleetbench cpu --duration 900s` (~15 min) before worker-runner starts, so generic-worker is intentionally not up during the run. The hourly gw_exe_check task can fire mid-benchmark once uptime passes its 15-minute grace period, see no generic-worker process, and escalate to reboot/PXE reimage. maintainsystem-hw.ps1: set MOZ_FLEETBENCH_RUNNING (run's UTC start time) at Machine scope around the benchmark call, cleared in a finally so it is always removed even on error. gw_exe_check.ps1: read the marker live from the registry ([Environment]::GetEnvironmentVariable(..,'Machine'), not the possibly-stale process env block) and skip the check while it is fresh. The marker is a timestamp with a 30-min staleness cap, so a crash/reboot mid-run cannot silently disable the watchdog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Incorporated into the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fleetbench-based PSU / firmware-throttle detector for the NUC13 hardware fleet.
Split out of #1235, which combined this with the Windows Defender hardening
(RELOPS-2396); that half is now in a separate PR.
Hardware-only: gated to the datacenter maintain-system path + the
hardware_observabilityprofile, so cloud/VM workers are unaffected.
What it does (per hardware node)
win_fleetbenchmodule installs the version-pinned collector toC:\fleetbenchplus per-hardware baselines (fleetbench_baselines.json), wired viahardware_observability. Version pinned indata/os/Windows.yaml.maintainsystem-hw.ps1runsfleetbench cpu --mode quick --duration 900s --jsonbefore worker-runner starts (once post-bootstrap, then at most once per 72h). 900s
self-warms the node so PSU/thermal throttling actually surfaces — a short cold-boot run
can false-pass.
(nuc13: min-floor ≥75 / mean ≥100 / tputCV ≤25 = GOOD; min<50 OR cv>40 OR mean<95 = BAD).
Hardware type identified in-script from
Win32_ComputerSystem.Model; unknown hardwarelogs and does not error/block.
fleetbenchandfleetbench_variancesurfaceverdict + metrics to Marlin (matching Marlin services live in mozilla-it/marlin).
Files
New:
modules/win_fleetbench/{manifests/init.pp,files/run_fleetbench.ps1,files/fleetbench_baselines.json},modules/win_nsclient/files/check_fleetbench.ps1,modules/win_nsclient/files/check_fleetbench_variance.ps1Modified:
data/os/Windows.yaml,hardware_observability.pp,win_nsclient/manifests/init.pp,win_nsclient/templates/nsclient.ini.epp,win_scheduled_tasks/files/maintainsystem-hw.ps1(fleetbench functions + the
Invoke-FleetbenchCheckcall beforeStartWorkerRunner).🤖 Generated with Claude Code