Quarantine unhealthy vGPU VFs via a persisted health store

[yummybomb]

Summary

Bottom half of the wedged-VF work (detection lands separately on top, in #435). This PR adds the quarantine machinery: a persisted VF health store plus the placement, admission, and API changes that act on it. Nothing in this PR reports failures automatically yet — the store is fed by the guest-side detection in #435, and by operators editing the state file per the runbook.

• VF init failures are tallied in <data-dir>/gpu/vf-health.json per instance assignment; a VF is quarantined once failures accumulate from gpu.vf_quarantine_threshold distinct assignments (default 2). An exact-assignment success report clears the match and older tallies, and rescinds that assignment's quarantine.
• Quarantined VFs are excluded from placement and advertised profile availability. Cards with quarantined VFs are deprioritized, and selection among equivalent free VFs is randomized so one VF cannot capture every placement.
• GET /resources reports allocatable_slots and quarantined_slots; GPU admission gates on the allocatable count.

Safety and failure handling

• An unreadable or invalid state file fails closed: mutations are refused, placement and advertised availability are disabled, and loads are retried after repair.
• State writes fsync the temp file before the rename and the directory after, so a host crash cannot silently drop a quarantine.
• State write failures before rename roll back in memory. Failures after rename retain the visible state and keep placement closed until a later report durably re-persists it.
• Quarantine only removes capacity — it never touches a running instance.

lib/devices/GPU.md documents the store semantics, draining the parent GPU, the SR-IOV recovery cycle, and clearing quarantine state.

Testing

Passed:

go test -race ./lib/devices ./lib/paths ./lib/resources ./cmd/api/config
go vet ./lib/devices ./lib/resources ./lib/paths ./cmd/api/config ./cmd/api/api

The full lib/instances suite was also attempted; TestCreateInstanceWithNetwork fails in this environment on both this branch and its base (it boots a real VM image), so it is environmental, not introduced here. The stacked detection PR's tree is byte-identical to the previously reviewed and tested head of #435.

---

Note

Medium Risk
Changes vGPU placement and admission on a fail-closed persisted store; misconfigured or corrupt vf-health.json can block new vGPU placements until fixed, though running VMs are not touched.

Overview
Adds persisted VF health (gpu/vf-health.json) and wires it through vendor-VFIO placement, capacity reporting, and GPU admission so wedged VFs stop capturing every new instance.

Guest init failures are tallied per distinct instance assignment; after gpu.vf_quarantine_threshold (default 2) the VF is quarantined—dropped from placement and profile availability, with parent GPUs deprioritized and random choice among equivalent free VFs. ReportVFInitFailure / ReportVFInitSuccess update the store (success clears only a matching assignment and can rescind that assignment’s quarantine). Invalid or unreadable state, or a latched persist failure, fails closed on placement and allocatable counts until repair.

GET /resources gains required allocatable_slots and quarantined_slots; reservation/validation gates on allocatable slots (not total_slots - used_slots). Device manager loads health at startup; config documents vf_quarantine_threshold.

Docs in GPU.md cover store semantics, manual drain/quarantine, and SR-IOV recovery (including DCGM quiesce). Automatic failure reporting is expected from a stacked PR—this change is the quarantine machinery only.

^{Reviewed by Cursor Bugbot for commit ce05d4e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at db6a6c7 (pushed)

go get github.com/kernel/hypeman-go-staging@db6a6c7eeab5808b729eef0d5383e0f13d89cf3a

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

230 files generated at f9e149e (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at 8f55018 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	bd_76DdwqSX-lucky-parsley
Timestamp	2026-08-27T20:45:12.782Z
stlc	8413509
Spec hash	3652069ab265
Config hash	55e15f6f4434

---

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-27 20:45:41 UTC

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2bea9f0. Configure here.}