[DRAFT] win116424h2hwbake: bake role for the Windows HW WIM pre-bake pipeline - #1297
Draft
markcor wants to merge 28 commits into
Draft
[DRAFT] win116424h2hwbake: bake role for the Windows HW WIM pre-bake pipeline#1297markcor wants to merge 28 commits into
markcor wants to merge 28 commits into
Conversation
Contributor
Author
|
Pulled in the fleetbench PSU/firmware-throttle detector from #1263 (RELOPS-2402) — cherry-picked both commits ( |
Trimmed copy of win116424h2hw that omits the four deploy-time/ machine-specific/hardware profiles (windows_worker_runner, microsoft_kms, nuc_bios, nuc_management) so the stable catalog can be baked into a golden install.wim on a reference VM, Sysprep-generalized, and captured. The omitted profiles run at first boot on the real NUC via the full win116424h2hw role. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors win116424h2hw.yaml without generic_worker.client_id (worker registration is excluded from the bake role). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eshness - Exclude scheduled_tasks from the bake role. Those are runtime/operational tasks (maintain_system startup loop, self_redeploy_check -> PXE redeploy, gw_exe_check, task-user logon) that could fire during the bake or on the generalized image's first boot before the deploy-time run. The full win116424h2hw role registers them at deploy time (same rationale as excluding windows_worker_runner). - TODO comment on google_chrome: baked Chrome can go stale between bake and deploy; ensure it's refreshed before the first worker-runner start. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove hardware_observability (win_nsclient monitoring agent; looks up marlin_pw) and windows_datacenter_administrator (local admin password from win_adminpw) from the bake role. Both are deployment-time identity/config tied to the datacenter and must not be baked into a generalized image; the full win116424h2hw role applies them at first boot. Consequence: the bake role now references NO Vault secrets, so the bake needs only an empty placeholder vault.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…loy-time) Chrome is the only chocolatey consumer in the role. Baking it (a) requires the chocolatey provider, which is not functional in the bake's single puppet pass (provider suitability is decided at run start, before choco installs), and (b) goes stale between bake and deploy. Install Chrome at worker deployment instead: keeps it current and removes chocolatey from the bake entirely. Everything else (7-Zip, MozillaBuild, Git, etc.) installs directly from the assets blob, not choco. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenSSH server + the winaudit SSH key on the Administrator account are datacenter access/identity, set up at deploy. They also can't apply cleanly on a generalized bake image: the Administrator profile dir doesn't exist, and enabling sshd in a single pass races the OpenSSH install. Defer the whole ssh profile to deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… disable Support the wim-packer image bake (no D: secrets drive, single puppet pass): - facts_win_github.ps1: fall back to $env:custom_win_github_pat when D:\secrets\pat.txt is absent, so the bake can supply the GitHub token as a build-scoped env var (from the pipeline / GHA GITHUB_TOKEN) instead of a secrets drive. D: still takes priority for datacenter/MDC1 deploys; never written to disk in the bake. - download_tooltool.ps1.epp: only send the GitHub Authorization/API headers when a PAT is present. An EMPTY Bearer on a public raw.githubusercontent.com URL returns 404 (not 401), which broke the bake; also fixes production re-applies with a momentarily empty fact. With a real PAT, behavior is unchanged. - wsearch/disable.ps1: make it idempotent (guard the takeown/rename on the indexer still existing). Previously it failed on every re-apply after the first — puppet re-applies on a schedule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The removal loop logged "removing AppX match: <X>" for every key in the list
unconditionally - so a WIM already debloated in the bake looked like it was
removing ~48 apps on every run (misleading; the real removals are silent inside
try{}). Snapshot provisioned (DISM, survives AppXSvc-disabled) + installed once,
then per app: remove if actually present, otherwise log "not installed, skipping".
Makes the log reflect reality and avoids spawning removal jobs for absent apps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…anary CompareConfigBasic's refresh-rate check reimages when CurrentRefreshRate != 60. nuc13-160 is headless/on a KVM and reports 1, so a SUCCESSFUL deploy immediately Set-PXE'd into a reimage loop. Comment out the reimage (keep logging the value) so the canary stays up to troubleshoot. TODO: restore / make headless-aware before master. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CompareConfigBasic downloads pools.yml from main and reimages if the node's ronin hash / image dir don't match. The RELOPS-2487 canary deploys from the nuc-wim-pipeline branch (hash + image live there), so checking against main's relops1213 (edef633 / older image) was a permanent false "config mismatch" -> Set-PXE loop right after a good deploy. Point the drift-check URL at the dev branch. TEMP: revert to main before master. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-enable the bug-1910123 check (reimage when CurrentRefreshRate != 60) that was disabled for the RELOPS-2487 canary. A NUC that reports refresh rate 1 has fallen back to the Microsoft Basic Display Adapter (the real Intel GPU driver isn't loaded) - a bad environment for our workers, so reimaging it is correct. The fix for the pre-baked-WIM path is to inject the Intel drivers into the WIM (win-hw-wim drivers.inject) so the node comes up at 60, not to soften this check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g (RELOPS-2487) The pre-baked node still reports refresh rate 1 (Intel GPU driver not active) even after baking the NUC13 driver pack, so the strict check reimage-loops the canary. Comment out Set-PXE (keep logging the observed rate) to stop the loop while we diagnose why the Intel display driver isn't loading. Restore once it reports 60. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esh rate Refresh rate was a fragile proxy: these NUCs run through a Raritan KVM whose EDID advertises 64 Hz (healthy != 60), while a driverless node reports the sentinel 1. Check the real signal instead: is the display still on the generic Microsoft Basic Display Adapter (no Intel GPU driver) vs a real Intel GPU. This is KVM-session- independent (the driver binds whether or not a Raritan session is open, and it's closed during test runs). Set-PXE stays commented while we get the Intel graphics driver into the bake; restore it once the real GPU is in use. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d in) The golden WIM now bakes in the Intel graphics driver (proven on nuc13-160: iGPU binds Intel Iris Xe @ 60 Hz), so a node on the generic Microsoft Basic Display Adapter is a genuine failure. Restore Set-PXE in the adapter check (suppressed during driver troubleshooting). Keeps the adapter-based detection (not the fragile refresh-rate proxy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a pre-baked (already debloated) image nearly every key on the ~48-app list is absent, so this DEBUG line was ~48 lines of pure noise per run. Skip silently when a package isn't present; keep logging actual removals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split out of PR #1235 (combined RELOPS-2396 branch) into fleetbench-only. Per NUC13 hardware node, before worker-runner starts: - win_fleetbench module installs the version-pinned collector to C:\fleetbench plus per-hardware baselines, wired via the hardware_observability profile. - maintainsystem-hw.ps1 runs `fleetbench cpu --mode quick --duration 900s --json` once post-bootstrap then at most once per 72h; 900s self-warms the node so PSU/thermal throttling surfaces (a short cold-boot run can false-pass). - Evaluates GOOD/BAD/MARGINAL/UNKNOWN vs the locked per-hardware baseline, plus drift vs the node's first recorded run. - NSClient++ checks `fleetbench` and `fleetbench_variance` surface verdict + metrics to Marlin (Icinga2/Grafana). Hardware-only: gated to the datacenter maintain-system path + hardware_observability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 2aac40e)
Invoke-FleetbenchCheck runs `fleetbench cpu --duration 900s` (~15 min) before worker-runner starts, so generic-worker is intentionally not up during the run. The hourly gw_exe_check task can fire mid-benchmark once uptime passes its 15-minute grace period, see no generic-worker process, and escalate to reboot/PXE reimage. maintainsystem-hw.ps1: set MOZ_FLEETBENCH_RUNNING (run's UTC start time) at Machine scope around the benchmark call, cleared in a finally so it is always removed even on error. gw_exe_check.ps1: read the marker live from the registry ([Environment]::GetEnvironmentVariable(..,'Machine'), not the possibly-stale process env block) and skip the check while it is fresh. The marker is a timestamp with a 30-min staleness cap, so a crash/reboot mid-run cannot silently disable the watchdog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 74e8909)
…there is no work disable_appxsvc writes AppXSvc Start=4 AND registers the at-startup task \Hardening\Hard-Disable-AppXSvc. During a WIM bake both are captured into the golden install.wim, which silently defeats the ref/ref-alpha pool exemption added for bug 2013985: that exemption only makes puppet DECLINE to disable the service at deploy time, it never re-enables it. Deployed ref-alpha nodes therefore came up with AppXSvc disabled, the HEVC/AV1/VP9/WebMedia extensions stayed provisioned but never registered for the per-task user, and Firefox fell back to 'ffvpx video decoder (RDD remote)' instead of the WMF hardware decoders -- failing mochitest-media-mda-gpu (HEVC unsupported, VP9/AV1 not hardware decoded, hevc/mkv playback timeouts). Skip disable_appxsvc when custom_win_role ends in 'bake' so the image ships AppXSvc Manual and each pool's deploy-time run makes the call, which is what the exemption always assumed. Leaving AppXSvc enabled in the image has a cost: wsappx can then run, so Wait-AppxIdle in win_uninstall_appx_packages.ps1 is no longer instantly-idle and would block up to 600s on every deploy for zero work on an image the bake already debloated. Add a cheap pre-check (two enumerations, no waiting) that skips both the wait and the removal pass when nothing on the removal list is present. The app-key list is hoisted to script scope so the pre-check and the removal share one list. Deliberately NOT done as a puppet-level reorder of disable_appxsvc ahead of the removal: most of the fleet still runs the MDT-built win11-24H2-NUC-01-16-2025 image and relies on the deploy-time removal, which a reorder would silently stop. The pre-check is inert on those nodes because keys are present. The shortcut is gated on the provisioned enumeration having actually succeeded -- that view is DISM-level and does not need AppXSvc, so a failure there means we cannot see the image and must not mistake it for "already debloated".
…ng on a generalized name Reverses the approach in b64eb0b. That commit stopped the bake disabling AppXSvc so the image would ship it Manual. It fixed the codecs but changed sysprep's behaviour: the new WIM gets a full generalize, so the first-boot specialize pass regenerates a random WIN-xxxxxxxx into ActiveComputerName. maintainsystem-hw then looked the node up under that name, missed, and Set-PXE'd -- re-imaging into a fresh random name, forever. The bake never needed to change. The HEVC/AV1/VP9/WebMedia packages are PROVISIONED into the image, which is DISM-level and needs no AppXSvc; only the per-user registration at first task_* logon does. So: stage at bake, register at deploy. - disable_services.pp: the bake disables AppXSvc again (restoring the known-good sysprep path). ref/ref-alpha now get an ACTIVE re-enable rather than an exemption -- merely declining to disable is useless on a pre-baked WIM, which arrives with Start=4 and the \Hardening\Hard-Disable-AppXSvc startup task already in place. - win_disable_services::enable_appxsvc (new): removes the hardening task FIRST (it runs at startup and would otherwise re-disable the service on the next boot), then sets Start=3/Manual -- matching stock Windows and the working MDT reference nodes. - maintainsystem-hw.ps1 CompareConfigBasic: resolve the node from the PERSISTENT ComputerName instead of $env:COMPUTERNAME (ActiveComputerName), which is the value specialize corrupts. Where the two disagree, take an ordinary reboot to activate the pending rename -- bounded to 2 attempts so a name that never activates degrades into a loud log rather than an endless cycle. And never Set-PXE on a name still matching ^win-: that means "identity not established yet", not "wrong node", and re-imaging over it is what made the loop unbreakable. Observed on nuc13-158: ComputerName=NUC13-158 but ActiveComputerName=WIN-D81J5HC82S0, with Tcpip Hostname/NV Hostname both correct -- so OS-deploy's offline rename worked and only the active name was clobbered.
win_uninstall_appx_packages.ps1's Write-Log emitted the message with Write-Output,
which puts it on the PIPELINE. Any Write-Log call inside a function that returns a
value therefore appends the log line to that function's return.
That went live with the win11-24h2-hw-20260820-235936 WIM. The bake disables AppXSvc
again, so at deploy Get-AppxPackage throws, Get-AppxSnapshot's catch logs a WARN, and
the function returns Object[] { '<warning text>', <hashtable> }. Binding that to the
[hashtable] $Snapshot parameter threw:
Cannot convert the "System.Object[]" value ... to type "System.Collections.Hashtable"
so the script exited 1, puppet exited 6, and bootstrap's exit-6 branch ran Set-PXE.
Every node re-imaged onto the same image and failed the same way - a hard loop, hit on
all five canary nodes (t-nuc12-002/003, nuc13-158/074/115).
Write-Host instead: still rendered to stdout for Puppet's logoutput and still captured
by the transcript, but never on the pipeline. The duplicate UserInteractive Write-Host
block is now redundant and removed.
Note the same Write-Output-in-Write-Log pattern is still present in win_enable_appxsvc.ps1
and win_disable_appxsvc.ps1. Neither logs from inside a value-returning function today,
so neither is broken, but both are one edit away from this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…anch too maintainsystem-hw.ps1's CompareConfigBasic was repointed at the nuc-wim-pipeline pools.yml back in August, but self_redeploy_check.ps1 - a SECOND, independent drift checker installed on every datacenter Windows node - still read main. The canary pools carry their hash and image on the dev branch; main holds only the `dev:` trigger and a stale hash (a22e7ac). So self_redeploy_check compared the node's GITHASH against main's and logged "Git Hash MISMATCH!" on every run, while CompareConfigBasic on the same node logged "Git Hash Match: f1eeb1f" a minute later. Observed on all five canary nodes: a mismatch sets $SETPXE, and with no active task that path is Set-PXEPendingFlag -Clear + Restart-Computer, so the nodes were rebooting roughly hourly (t-nuc12-002 18:53, 19:06, 19:49, 20:00; nuc13-158 18:31, 19:32, 20:01). Had a task been running instead, the other branch calls Set-PXE and re-images the node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… image
A node deployed from the pre-baked golden WIM came up with Virtualization-Based
Security entirely off, while the MDT-built production image runs VBS with both
Credential Guard and HVCI. HVCI runs the kernel under a hypervisor and taxes
syscalls, memory management and I/O, so the two populations were not measuring
the same machine - baked nodes looked faster than the fleet they represent.
Measured on nuc13-006 (production) against nuc13-115 (baked); the delta was
exactly two registry values:
HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard
EnableVirtualizationBasedSecurity = 1
HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity
Enabled = 1
Nothing else was needed, which is worth recording because both are easy to get
wrong:
- Credential Guard needs no key of its own. Production shows
SecurityServicesRunning 1,2 with no CredentialGuard scenario key and
LsaCfgFlags unset - it is default-on for Windows 11 Enterprise once VBS is up.
- No Hyper-V optional feature is required. Production has every
Microsoft-Hyper-V-* feature Disabled yet reports HypervisorPresent=True;
Windows loads the hypervisor for VBS on its own.
New win_device_guard module, driven from hiera via a device_guard profile, wired
into the bake role and the two canary roles. Both nodes already report
AvailableSecurityProperties 1-8 and Secure Boot on, so the hardware is ready.
Takes effect on the next boot, which the deploy gets for free (bootstrap reboots
after the puppet run). In the bake VM the values are written but VBS will not
activate - nested virtualisation is not exposed to the packer guest - which is
expected: the WIM carries the setting and it activates on real hardware.
Not added to win116424h2hw: production already has this from its image, so
puppet-enforcing it there is a separate, fleet-wide decision.
puppet-lint clean (5.1.1, repo flag set).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…GraphicsSoftwareService Production MDT nodes run IntelGraphicsSoftwareService; nodes built from the pre-baked WIM do not. It is not a driver service - it ships inside the AppUp.IntelArcSoftware MSIX. Under DCH the graphics *software* is a separate Store-delivered package that Windows Update fetches as a driver companion app, and our images have WU disabled by design, so the MU-catalog driver cab (INF only) can never supply it. Intel's full installer does carry it, as Resources/Extras/IntelGraphicsSoftware_26.18.2353.2_Release.exe - confirmed by scanning gfx_win_101.7088.exe. Note that win_packages::drivers::intel_gfx passes --noExtras, which is exactly the flag that skips that folder; this class does not pass it. Package uploaded to hardwareimaging resources/drivers/gfx_win_101.7088.exe (771,672,256 bytes, sha256 6c2cd36a5c68cf3aa8403073e5ed4592b8ee4ba5190f5776e483d974ead5a2f8). Stage at bake, install at deploy - and it has to be that way, because the blob is Entra-only: an anonymous GET to hardwareimaging returns 409, while the deploy-time mirror roninpuppetassets returns 200. The bake build host holds a managed identity and can azcopy the installer in; a deployed NUC has no Azure identity and cannot. So this class never downloads. It is idempotent and self-selecting: bake - service absent + installer staged locally -> runs it into the image deploy - service already present from the WIM -> no-op, starts it if stopped Missing installer is a WARN, not a failure, so a node on an older WIM is not bricked by a catalog error. The script also reports whether the package ended up provisioned image-level, since a per-user-only install would be stripped by sysprep /generalize. STILL REQUIRED (worker-images, not this repo): prepare-base-vhdx.ps1 must stage the installer to C:\bake\extras\ during the bake, the same route the driver cabs take. Until that lands this class is a no-op everywhere. puppet-lint clean; script AST_OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion Production is not uniform. The NUC13 pool (win11-64-24h2-hw) has IntelGraphicsSoftwareService; the NUC12 reference pool (win11-64-24h2-hw-ref) does not - measured on t-nuc12-004/005/006, all three report the service absent. The goal is to replicate production, so ref-alpha (NUC12) must not have it either. One golden WIM serves both platforms and the bake provisions the MSIX image-wide, so the NUC12 pools cannot simply decline to install it: by the time puppet runs it is already in the image. They need an ACTIVE removal, the same shape as win_disable_services::enable_appxsvc actively undoing the baked AppXSvc disable. So win_intel_graphics_software gains $ensure: present (default, NUC13 + bake) - install if the service is absent, else start it absent (NUC12 roles) - remove the provisioned and installed package The exec guard is the mirror image of $ensure, so both directions are idempotent. $ensure is a profile PARAMETER rather than a plain hiera lookup because the discriminator is the role, and the Windows hiera hierarchy has no role level: win_hiera.yaml keys its per-worker level on custom_win_gw_workertype, and that fact is EMPTY on these hardware pools - HKLM\SOFTWARE\Mozilla\ronin_puppet\workerType is unset on nuc13-115, t-nuc12-002 and t-nuc12-004 alike, so that level never matches anything. win116424h2hwrefalpha therefore declares the profile with ensure => 'absent'; NUC13 roles take the default. Removal is best-effort and logs each step. Get-AppxPackage enumeration is expected to fail wherever AppXSvc is disabled; the provisioned removal is the one that matters on a freshly deployed node with no extra user profiles. puppet-lint clean; script AST_OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…01.7088
Groundwork for moving the NUC12 reference pool off the 31.0.101.3729 its MDT image
shipped. ref and ref-alpha are currently a whole major driver version apart
(31.0.101.3729 vs 32.0.101.7085), which makes the reference pool not comparable to the
pool it is the reference for.
This class was commented out of every role and could not have worked if enabled:
1. it staged the installer to <systemdrive>\intel\gfx.exe but executed
C:\Windows\Temp\gfx.exe - a path nothing ever wrote to;
2. its guard read $facts['custom_display_adpater'] (sic), a fact defined nowhere in
this repo, so it was always undef and the guard always true;
3. the file resource fetched a ~750 MB installer on every run whether or not the driver
already matched.
Now: one guarded exec that downloads, installs and cleans up, skipped entirely when
Win32_VideoController already reports the target version. A node already on target
transfers nothing.
--noExtras is kept and is load-bearing: it skips Resources/Extras, where
IntelGraphicsSoftware_<ver>_Release.exe lives. Production's NUC12 reference pool does not
have IntelGraphicsSoftwareService and must not gain it, so this class installs the driver
and nothing else. The NUC13 pools get the software separately via
win_intel_graphics_software, which runs the same installer without that flag.
hiera windows.driver.gfx.version 101.4255 -> 101.7088.
NOT YET ENABLED ANYWHERE. roles_profiles::profiles::intel_drivers is still commented out
in win116424h2hw, win116424h2hwref and win116424h2hwrefalpha, and it is BLOCKED on
hosting: the class pulls from the public roninpuppetassets mirror, which has only
gfx_win_101.4255.exe - 101.7085 and 101.7088 both 404 there. Deployed NUCs have no Azure
identity and cannot read our Entra-only hardwareimaging account, so the installer has to
be on the public mirror before ref can move.
puppet-lint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IntelGraphicsSoftwareService is present on MDT production NUC13 nodes and absent on
every pre-baked one. The install was wired into the bake role and has never once
worked; this moves it to the deploy-time roles, where it is proven to work.
Two independent reasons the bake cannot do it:
1. Intel's installer returns rc=1008 in the packer guest. On real NUC13 hardware
the same installer, driven by this same script, returns rc=1001 and leaves
IntelGraphicsSoftwareService Automatic/Running (measured on nuc13-115,
2026-09-01). The GPU-less Hyper-V build guest is the obvious difference.
2. Even on success it would not survive capture. Intel lays the
AppUp.IntelArcSoftware MSIX down as a per-user install, not a provisioned one -
the script's own check reported `provisioned packages matching: 0` - and
sysprep /generalize strips per-user AppX from the image.
So the WIM now ships the INSTALLER rather than the installed software:
- installer_path C:\bake\extras\gfx_win_*.exe -> C:\extras\gfx_win_*.exe.
worker-images prepare-base-vhdx stages extras.files there instead of under
C:\bake, which sysprep-generalize.ps1 deletes wholesale before capture (it holds
the SYSTEM puppet helper embedding the build GitHub token, so that scrub stays).
That deletion is the entire reason deployed nodes found no installer and silently
took the best-effort skip branch.
- win116424h2hwbake drops the profile, so the bake stops burning time on an install
that cannot succeed.
No change to the deploy-time contract: perf-debug and the other NUC13 roles keep
ensure => present, ref-alpha keeps ensure => absent, and the exec is still guarded on
the service, so a node that already has it does nothing.
Also reworded the provisioned-package log line. Zero provisioned packages is the
EXPECTED result of a per-user install and is not a fault; the service is the success
signal. The old wording warned about sysprep stripping the WIM, which is meaningless
in the deploy-time context this now runs in.
Requires a re-bake: the current golden WIM has no C:\extras.
Ref: RELOPS-2487
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on the 2026-09-08 deploy of win11-24h2-hw-20260908-172915: Intel's installer
takes 2m46s-3m03s (nuc13-115/074/158) and ran inside the deploy puppet apply, roughly
doubling its ~2.5-2.9 min. It is most of the NUC13-vs-NUC12 spread on that deploy -
16-18 min against 12-13 min for a pool that never runs it.
Nothing at deploy time needs the service, so blocking the node from reporting ready on
it buys nothing. win_intel_graphics_software now drops a marker holding the installer
path, and the first maintainsystem run after deploy consumes it.
present -> marker only, no exec. maintainsystem-hw.ps1 installs it.
absent -> unchanged, still inline. Removal is a fast no-op on a node that never had
the service, and a NUC12 pool must not be left carrying it even briefly.
The marker doubles as the ensure discriminator: only NUC13 roles get one, so
maintainsystem needs no role logic of its own. Its content is the installer path, so
hiera stays the single source of truth. maintainsystem deletes it BEFORE the attempt -
a failed or crashed install must not retry three minutes on every boot. The task is
boot-triggered and runs as SYSTEM, which the installer needs.
Deploy-time change only, no re-bake: the WIM already carries C:\extras.
KNOWN TRADE, worth stating for a perf pool: the node now reports ready ~3 min sooner,
but generic-worker can claim a task while the installer is still running on the first
boot after a re-image. Treat the first task after a re-image as suspect, or wait for
"Intel Graphics Software done" in the MaintainSystem event log before trusting numbers.
Ref: RELOPS-2487
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ntainsystem" This reverts commit ccee773. The deferral does not actually save the three minutes, it relocates them. The work is irreducible either way; all that changes is whether the node is blocked during it or busy during it. inline puppet blocks ~3 min -> registers later -> the post-puppet reboot has already happened, so the first task runs on a settled machine deferred registers ~3 min sooner -> maintainsystem installs at boot, concurrently with worker-runner starting, so the first task after a re-image can land on a machine mid-install For a general CI pool the first framing is the right one and the deferral is a real win. perf-debug and ref-alpha exist to produce trustworthy numbers, and there it buys close to nothing: "3 min not ready" becomes "3 min ready-but-unreliable", plus a standing requirement to remember to discard the first task after every re-image. Bad trade on the two pools currently being used to chase a 2.8% difference. Removing the 3 min for real needs the AppUp.IntelArcSoftware MSIX provisioned into the WIM so nothing installs at deploy. That is blocked on sourcing, not on hardware - Add-AppxProvisionedPackage is image-level and needs no GPU, which is how the media codecs already get baked - but no standalone .msixbundle exists to provision: a prod node keeps only the unpacked signed staged directory, winget msstore returns nothing, Intel's site blocks scripted access, and gfx_win_101.4255.exe does not contain it. The one unexplored route is the copy the MDT share holds, which is where production got 26.26.2459.0. Back to a312287, which is verified working on all five canary nodes: perf-debug GFXSVC Running/Automatic, ref-alpha correctly ABSENT, ERRDEV 0 on both. Ref: RELOPS-2487 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
win116424h2hwbake— a trimmed variant of thewin116424h2hwhardware role, used only at image-bake time to pre-apply the stable, expensive catalog (notably the ~30-min AppX removal) into a golden WIM. The baked WIM is then deployed to NUCs, cutting deploy-time Puppet work.manifests/roles/win116424h2hwbake.pp— the bake role.data/roles/win116424h2hwbake.yaml— its Hiera data.How it differs from
win116424h2hwExcludes the profiles that must run at deploy time on real hardware / per-worker, not at bake time:
windows_worker_runner(no worker registration / secrets baked in),microsoft_kms,nuc_bios,nuc_management.So the bake applies the machine-generic, stable catalog; identity, worker registration, KMS, BIOS and datacenter management stay in the normal deploy-time role.
Usage
Consumed by the wim-packer bake (
bake-bootstrap.ps1in worker-imagesprovisioners/windows/win-hw-wim/): it seedsHKLM:\SOFTWARE\Mozilla\ronin_puppetwith this role, generatesnodes.pp, and runspuppet applyinside the build VM before Sysprep + capture.🤖 Generated with Claude Code