Skip to content

fix: mark all devices sharing a physical GPU unhealthy on Xid errors - #1930

Open
tryuuu wants to merge 1 commit into
NVIDIA:mainfrom
tryuuu:fix/unhealthy-report
Open

fix: mark all devices sharing a physical GPU unhealthy on Xid errors#1930
tryuuu wants to merge 1 commit into
NVIDIA:mainfrom
tryuuu:fix/unhealthy-report

Conversation

@tryuuu

@tryuuu tryuuu commented Jul 17, 2026

Copy link
Copy Markdown

Fixes #1929

checkHealth kept only one *Device per physical GPU UUID in parentToDeviceMap, so devices sharing a physical GPU overwrote each other. As a result an Xid event marked at most one device as unhealthy. With replication (time slicing or MPS) the other replicas of the broken GPU stayed schedulable, and with MIG the event could even be dropped entirely because the targeted instance was never stored in the map.

This PR changes the map to map[string][]*Device and fans each Xid event out over all devices under the event UUID, so every affected device is reported as unhealthy.

@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@tariq1890

Copy link
Copy Markdown
Contributor

Thank you @tryuuu for your contribution. On an initial glance, this looks fine to me. Let me run some more tests against this branch and get back to you sometime next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Health check misses devices that share a physical GPU

2 participants