Fix HTTP connection reuse in engine health probes

[Psych0h3ad]

Description

Periodic engine health checks close HTTP response bodies without reading them. With Go 1.26.7, this prevents the HTTP/1 transport from reusing those connections and opens a new connection for every poll. Draining ordinary responses before closing reduces unnecessary connection churn while PAIR monitors Ollama and LM Studio.

The regression tests reproduce 32 connections for 32 polls before the fix and one connection for the same 32 polls after it, for both Content-Length and chunked responses across all three health-check paths.

Release intent

Changelog title

Fix HTTP connection reuse in engine health probes

Changelog body

Periodic engine monitoring now reuses HTTP/1 connections instead of opening a new connection for every health probe.

Bumps

• services: patch
• nvpair-cluster-manager: none
• nvpair-engine-manager: patch
• nvpair-errors: none
• nvpair-job-scheduler: none
• nvpair-manual-nodes: none
• nvpair-node-info: none
• nvpair-node-scanner: none
• nvpair-node-settings: none
• nvpair-proxy: none
• nvpair-tui: none
• nvpair-ui-broker: patch
• nvpair-workload-manager: none

Scope

• Drain response bodies before closing in the broker's Ollama and LM Studio checks and the engine manager's HTTP probe, including unsuccessful HTTP statuses.
• Limit each drain to 1 MiB and retain the existing broker client timeout of 2 seconds and engine-manager probe deadline of 3 seconds.
• Add tests for HTTP/1 connection reuse, unchanged status-based health decisions, bounded reads, and stalled-body deadlines.
• Bump nvpair-ui-broker from 0.40.2 to 0.40.3 and nvpair-engine-manager from 0.17.4 to 0.17.5.

No API, configuration, dependency, or desktop changes are required. This change addresses connection churn; it does not reset existing TCP state or establish a cause for any operating-system TCP reclamation failure.

Validation

After rebasing onto develop, the following passed on Windows amd64 with Go 1.27.0:

• go test ./... -count=1 from services/shared
• go test ./... -count=1 from services/nvpair-engine-manager
• go test ./... -count=1 from services/nvpair-ui-broker
• Focused connection-reuse, bounded-drain, and deadline regression tests in all three affected modules
• node scripts/spdx-headers.mjs — 1,023 files checked, with no missing, review-required, or unclassified headers
• git diff --check upstream/develop..HEAD

The original targeted validation also passed on macOS arm64 with Go 1.26.7 and CGO_ENABLED=0.

No live engine operation was required because the regression tests exercise the real net/http HTTP/1 client and server over net.Pipe without consuming TCP source ports.

Risk

Health results continue to depend on the HTTP status, including when a body read fails. A stalled response can now keep the check active until the existing timeout or deadline, and an oversized or incomplete body can still cause the connection to be discarded. The drain is bounded in size and time and discards content without logging it. No data migration or wire-format change is involved.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

[Noah-Tervalon-Nvidia]

Thank you for bringing this issue up. We're going to have a look and validate that everything looks good on it.

[kjlubick]

I think this is worth fixing, although I noticed on golang 1.27, the tests w/o the fix only had 2 or 3 open connections. This seems to be due to HTTP/1 Response.Body now automatically drains any unread content upon being closed, up to a conservative limit, to allow better connection reuse. For most programs, this change should be a no-op, or result in a performance improvement (https://go.dev/doc/go1.27).

I'll send a commit to the branch to make the change be a bit more succinct and align with internal standards.

[kjlubick]

Just to touch base here, we are working on getting our final batch of internal changes out into the public repo. Once those are in, we'll get this rebased (if necessary) and landed. Thanks again for your contribution!

[Psych0h3ad]

Just to touch base here, we are working on getting our final batch of internal changes out into the public repo. Once those are in, we'll get this rebased (if necessary) and landed. Thanks again for your contribution!

Yay! I was struggling with this when I installed for my personal AI bot brain Mac Mini😁

The base branch was changed.

[kjlubick]

We'll need to rebase this onto develop (which has had some major changes lately). I can try to do that this morning.