Add managed llama.cpp engine with Windows CUDA support

[nv-pgoode]

Changelog title

Managed llama.cpp engine

Changelog body

• Install and manage llama.cpp from PAIR, with models retained across runtime uninstall.
• Route llama.cpp requests through the unified proxy to any downloaded model; the engine loads it on demand.
• Prefer the official CUDA build on compatible NVIDIA systems (Windows x64, Windows ARM64 and Linux), with a recorded fallback when CUDA cannot be used.
• Record the accelerator every llama.cpp installation actually has (CUDA, ROCm, Metal, Vulkan, CPU) and show it on the engine row.
• Bound llama.cpp downloads by lack of progress instead of a fixed 30-minute budget; force-stop an engine that ignores the graceful stop.
• Refuse new inference requests while every eligible node reports itself busy instead of queuing them into a saturated engine.
• Report a streamed response the client received in full as completed even when the client disconnects right after the final chunk.

Bumps

• services: minor
• nvpair-cluster-manager: none
• nvpair-engine-manager: minor
• nvpair-errors: none
• nvpair-job-scheduler: minor
• nvpair-manual-nodes: minor
• nvpair-node-info: minor
• nvpair-node-scanner: minor
• nvpair-node-settings: none
• nvpair-proxy: minor
• nvpair-tui: minor
• nvpair-ui-broker: minor
• nvpair-workload-manager: none

Summary

Add managed llama.cpp using the existing Engine Manager, unified proxy and
engine-settings UI. No new service process, no parallel settings implementation,
and no llama.cpp-specific routing rule.

• Install the official llama app into owned runtime storage; start/stop it;
  download/import/load/unload/delete models; cancel model downloads.
• On Windows x64, stage the pinned official CUDA app and runtime archives when
  every detected NVIDIA GPU has compute capability 7.5 or newer, and require a
  working CUDA device probe before promotion. Otherwise retain vendor selection
  and record why CUDA was not used; no local CUDA Toolkit installation is required.
• On Linux, apply the same gate: NVIDIA GPUs at compute capability 7.5 or newer
  run the pinned installer restricted to its CUDA payload (SKIP_VULKAN=1 SKIP_ROCM=1), a CUDA0: device is required before promotion and the transfer
  is retried once; otherwise the unrestricted installer runs and the receipt
  records cuda_not_used. Non-NVIDIA hosts are unaffected. Windows ARM64
  installs the pinned CUDA archives directly; the upstream-latest attempt is
  opt-in (install.upstream_first).
• Record the verified accelerator: every install receipt (pair-install.json)
  carries source, acceleration_policy and devices from llama cli --list-devices; engine:status exposes them as acceleration/devices; the
  desktop engine row and the TUI show the backend (CUDA, ROCM, METAL, VULKAN, CPU).
• Bound install and model-pull transfers by lack of progress (10 minutes idle,
  6 hours total) rather than a 30-minute wall clock, so a slow but live link
  finishes and a stalled one fails with that reason. Progress is anything
  observable: the staging tree or model cache growing, downloader stderr, and
  on Windows the child's own I/O counters (PowerShell buffers each download in
  memory). llama.cpp downloads land inside the install stage, never on the
  system temp drive. Fifteen seconds after an ignored graceful stop, the whole
  process group is force-killed. The runtime promotion retries a rename that
  Windows denies while Defender still holds the freshly written binaries.
• Proxy follow-ups found by live validation: a node answering 429/503 enters a
  one-second busy hold and a new request whose every eligible owner is held is
  refused with a local 503 and Retry-After instead of being queued into the
  saturated engine; a streamed response the client received in full is
  reported completed even if the client closes right after the final chunk.
• Keep model files when uninstalling the runtime. Treat external runtimes as
  read-only rather than taking ownership of another installation.
• Use the existing port/settings flow: proxy 8080, engine 8081 by default.
• Load models on demand: the managed router runs serve --models-autoload as a
  fixed argument, so a request for a downloaded model loads it first (the first
  response waits for the load; the router keeps up to four models resident and
  evicts idle ones). Routing eligibility follows the advertised catalog, exactly
  as it does for Ollama and LM Studio; the loaded-model watcher remains for the
  UI's residency indicator and Load/Eject controls.
• Preserve client-disconnect and shutdown cleanup, retry/CORS behavior and
  exactly-once workload terminal delivery, with correct workload attribution.
• Add desktop and TUI engine/model controls, including producer-shaped tests
  for live proxy events and observed model residency.

Excluded: a per-engine switch to turn on-demand loading off (with
loaded-only routing), engine updating, GPU/OS telemetry or chart redesign,
expanded workload browsing, explicit inference-workload cancellation
controls/API, and a per-node pin for llama.cpp requests. Those are independent
follow-ups, not prerequisites for this integration.

Non-goals and known limits

• Existing installed runtimes keep their current backend. To move an existing
  Vulkan installation to the CUDA selection path, uninstall and reinstall the
  managed runtime; model files are retained.
• The vendor bucket has no aarch64/linux/cuda/110 payload (Jetson Thor); the
  gate records the missing build as cuda_not_used and the runtime lands on
  Vulkan. Receipts written before this change carry no acceleration_policy
  and the status shows nothing for them until reinstall.
• Engine Manager's manifest-only onboarding requirement is not met for three
  llama.cpp capabilities (staged official-app installer, on-disk model cache
  operations, router identity probe); they are engine-specific Go accepted only
  for the bundled manifest and are recorded in spec.md as a current gap.
• The broker's managed-facade coordinator for llama.cpp is a copy of LM
  Studio's with two policy differences; sharing it is left for the next
  managed engine.
• A cold load longer than the proxy's two-minute first-response budget fails
  that request while the router finishes loading; retry or load explicitly.
• Under concurrent load, the vendor's Vulkan build on an Intel Arc A770 stopped
  answering once; the proxy's response-header timeout and failover covered it.
  That is a vendor-build behaviour, recorded here so it is not mistaken for a
  routing fault.

Review First

Start with services/nvpair-engine-manager/manifests/llamacpp.json (fixed
serve --models-autoload, declared --host/--port controls), then
services/nvpair-proxy/proxy.go subscribedToNode (one eligibility rule for
every facade), services/nvpair-manual-nodes/manager.go probeLlamaCpp, and
the settings regressions in settings_llama_test.go and
services/nvpair-ui-broker/enginesettings_test.go. For the accelerator work:
llamaacceleration.go (classifier), llamalinuxcuda.go (Linux gate),
stall.go (progress-bounded contexts) and TestLlamaLinuxCUDAGate. For the
proxy follow-ups: the busy hold in proxy.go with busy_test.go, and the
completed-versus-cancelled classification around emitTerminal.

Validation

• All 13 GitHub checks pass on 81624e8 and again on a7391b0: desktop
  checks, SPDX, release intent, build-script checks on three operating
  systems, the full Linux Go component, cross-process and tooling suites, and
  six platform installer builds.
• Local Windows checks on 26271a0: all 15 Go modules build and vet. Full Engine Manager,
  broker, TUI, manual-node, cluster-manager, scheduler, workload-manager, errors,
  node-settings, shared and EAP suites pass; focused scanner/node-info and the
  cross-process external-listener refusal check pass. Network-test limits below.
• Desktop on 26271a0: 260 tests pass, 2 existing skips; typecheck, lint,
  dead-code, service contracts, build-script and SPDX checks pass. Go binaries
  and Electron production bundles build.
• Windows x64 CUDA on 26271a0: fresh managed CUDA installation passed.
  Four consecutive full GPU journeys passed after one earlier streaming-reload
  timeout. The passes cover cold/warm and streamed inference, unload/reload,
  unknown-model rejection, restart persistence, workload attribution and cleanup.
  The initial timeout remains recorded; its cause is not established.
• Windows CPU Desktop/TUI on predecessor f305c3b: actual Load/Unload,
  Start/Stop, proxy inference, completed-workload display and clean exit passed.
• Linux x64 NVIDIA on 26271a0: fresh managed install, model pull and first
  proxied inference passed, followed by warm inference, a completed long stream,
  streamed reload after unload, unknown-model rejection, restart persistence and
  five locally attributed completed workloads. Owned compute ran on the selected
  GPU; memory returned to baseline, both brokers exited normally, test ports and
  owned processes closed, and temporary state was removed. Unrelated GPU process
  identities were preserved. The earlier interrupted harness run is retained.
• Fleet validation of 81624e8 binaries on 13 lab hosts (2026-09-25), one full
  install/pull/serve/unload/restart/cleanup journey each: Windows ARM64 RTX
  Spark N1X (pinned CUDA), Snapdragon (CPU), Windows x64 RTX 3500 Ada, RTX 5090
  and RTX 5070 (CUDA), RTX 2070 on driver 566 (CUDA probe failed, recorded
  Vulkan fallback), Intel Arc A770+B580, Arc 130V and AMD 840M (Vulkan), Intel
  Mac (CPU), three GB10 DGX Sparks (CUDA), Jetson Thor (Vulkan). It also showed
  that Linux fell to Vulkan silently under download contention and that receipts
  could not tell CUDA from Vulkan; both are addressed by the follow-up commits.
• Follow-up commits (fefad2c through the head of this branch): Engine Manager,
  proxy and broker suites green, cross-vet for linux/arm64, linux/amd64,
  darwin/arm64, darwin/amd64 and windows/arm64, desktop 266 tests plus
  typecheck/lint/dead-code/service-contract checks, TUI tests, SPDX. Live
  re-validation with those binaries: Intel Arc Windows x64 and Arc 130V (Vulkan
  receipt and status), Apple M1 and M4 Max (Metal receipt after the classifier
  fix), Snapdragon Windows ARM64 (CPU build through the new ARM64 dispatch), RTX
  5090 and RTX 5070 over Wi-Fi (pinned CUDA archives finished after 35- and
  73-minute downloads that the previous 30-minute budget had cut off), a
  two-GPU RTX PRO 6000 workstation (both CUDA devices recorded and used), GB10
  DGX Sparks over slow links (restricted CUDA install with one observed retry
  after the vendor script landed on its CPU build). A 30 KB/s laptop exposed two
  false-stall cases fixed by a7391b0.
• Desktop application on CUDA (recorded, 2026-09-27): the desktop app built from
  this branch installed llama.cpp with the pinned CUDA archives on a two-GPU RTX
  PRO 6000 workstation through its own Install control, showed the CUDA pill with
  both devices, pulled a model, and paired seven lab nodes into a cluster from the
  GUI (RTX 5090, RTX 5070, RTX 3500 Ada, RTX 2070 on the Vulkan fallback, Arc
  A770/B580, Arc 130V, Snapdragon). Bursts of 24, 24, 24 and 48 concurrent
  768-token streaming requests: 144 of 144 completed at the client, every node
  served, four to six nodes running at once with their performance panels open,
  each completed job attributed to the node that ran it. The run surfaced the
  busy-hold, completed-versus-cancelled and rename-retry follow-ups above.
• GitHub Actions built unsigned installers for Windows x64/arm64, Linux
  x64/arm64 and macOS x64/arm64 on 81624e8; these are build results, not native
  hardware qualification for every platform.
• The separately committed desktop startup repair prevents late initial
  snapshots from replacing newer running/stopped events; regression checks pass.
• Test-only follow-up 81624e8 corrects an assertion that assumed Windows
  environment-name case folding on Unix. Canonical cache overrides remain
  rejected on every platform; lowercase aliases are additionally rejected on
  Windows.

Remaining Qualification

• The full proxy, scanner, node-info and cross-process suites passed on the
  isolated Linux CI runner. The local Windows gate used the focused network
  selection described above; this does not claim full Windows network-suite coverage.
• The Windows ARM64 direct-pinned CUDA path of fefad2c is covered by unit
  tests; the ARM64 RTX Spark lab host passed on 81624e8 through the fallback
  path and was reserved during the follow-up runs.
• Intel Mac re-validation of the follow-up commits was blocked by the lab
  host's endpoint agent, which kills the unsigned test binary during download
  regardless of commit; the Intel Mac passed on 81624e8 earlier the same day.
• The two-GPU Linux server (RTX 4090 x2) was offline throughout; multi-GPU CUDA
  was qualified on the two-GPU Windows workstation instead.
• The first Windows CUDA streaming-reload timeout was not reproduced in four
  later full runs or a targeted reload probe, but is not explained or erased.

Terve adapted the earlier integration onto the unified proxy. The earlier
llama.cpp work includes the proxy adaptation from #41; #18 and #36 provided
additional design/review context. This does not represent those PRs as merged
unchanged or claim authorship of their underlying shared PAIR infrastructure.

Checklist

• Contribution rules read; every commit has a matching DCO sign-off.
• Required tests and documentation updated; no new dependency added.
• Changed source/filenames/messages checked for private data and internal links.
• Release intent declared; bot-owned versions and changelog unchanged.
• Windows retained-CPU desktop/TUI functional journeys completed.
• Windows CUDA desktop functional journey completed (recorded, eight-node cluster).
• Remaining platform/GPU qualification above completed.

[Noah-Tervalon-Nvidia]

POC, so we're going to close the PR. Going to be bringing in llama.cpp with another targeted commit.