Skip to content

RELOPS-2402: Pin the production hardware pools to ronin f328b45d (fleetbench) - #961

Merged
markcor merged 1 commit into
mainfrom
relops-2402-fleetbench-prod
Sep 17, 2026
Merged

markcor merged 1 commit into
mainfrom
relops-2402-fleetbench-prod

Conversation

@markcor

@markcor markcor commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Picks up mozilla-platform-ops/ronin_puppet#1401 (fleetbench PSU/firmware-throttle detector), merged to master as f328b45d.

pool nodes image ronin hash
win11-64-24h2-hw 139 unchanged (win11-24h2-hw-20260908-172915) 47042c6b → f328b45d
win11-64-24h2-hw-ref 12 unchanged 47042c6b → f328b45d

Production's previous pin is an ancestor of f328b45d, so the only new content is fleetbench plus the dependabot grouping from #1402. No role edits were needed — fleetbench is wired into roles_profiles::profiles::hardware_observability, which every Windows hw role already includes (and win116424h2hwbake deliberately excludes, so nothing runs in the bake guest).

Validated on alpha first

worker-images#958 staged this on win11-64-24h2-hw-alpha (3 NUC13) and win11-64-24h2-hw-ref-alpha (2 NUC12). All five: bootstrap_stage=complete, puppet exit 2, worker-runner Running.

C:\fleetbench   fleetbench-0.4.6.exe (1,200,128)  run_fleetbench.ps1  fleetbench_baselines.json  results\
SHA256 on disk  8b98f0e72a8d1fff1756a4bc2c806a0fbc9583381218fed0c9d633446c7f07b4   == pinned hash
results\        2026-09-17T18-38-37Z_nuc13-018_cpu.json (26 MB) + fleetbench_status.json
NSClient++      check_fleetbench.ps1, check_fleetbench_variance.ps1 (+2 nsclient.ini entries)
GFXSVC          NUC13 Running/Automatic @ 32.0.101.7088 | NUC12 ABSENT @ 7085 (correct per role)

The collector downloaded, verified against the pinned checksum, and the benchmark actually ran and wrote an envelope — so this is not just "the files are present".

Two operational effects to expect fleet-wide

  • The benchmark blocks worker-runner start. Invoke-FleetbenchCheck runs the collector to completion before worker-runner, for 900s (15 min), once after bootstrap and then at most once per IntervalHours = 72. So each re-imaged node takes ~15 extra minutes before it claims its first task. Across a rolling 151-node re-image that is real, though it is a one-off per node and does not repeat until the 72h gate reopens on a later boot.
  • Result envelopes are 20-26 MB each and nothing prunes them. The only Remove-Item in the module clears the stderr file. At one envelope per 72h that is roughly 250 MB/node/month. Not urgent, and pruning needs care because the oldest envelope is the node's reference baseline for drop-off detection — so a naive "delete old files" would break the check. Worth a follow-up that keeps the oldest plus the most recent N.

Follow-up, not in this PR

The two alpha pools still pin the relops-2402-fleetbench branch @ c353d2b6. Now that #1401 is merged they should move to master @ f328b45d so they do not depend on a feature branch that can be deleted — that is a separate bump and another 5-node re-image.

🤖 Generated with Claude Code

Picks up ronin_puppet#1401 (fleetbench PSU/firmware-throttle detector), merged
to master as f328b45d. Production's previous pin, 47042c6b, is its ancestor, so
fleetbench plus the dependabot grouping in #1402 are the only new content.

Image is unchanged: win11-24h2-hw-20260908-172915.

Validated on both alpha pools first (worker-images#958, 5 nodes, all puppet
exit 2 / bootstrap complete): the v0.4.6 collector downloaded and its on-disk
SHA256 matches the pinned hash, the benchmark ran and wrote a result envelope,
and both NSClient checks plus their nsclient.ini entries are in place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@markcor
markcor merged commit 0341298 into main Sep 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant