[Bug]: High CPU Usage on Nontrivial MIG Configurations

[Arc676]

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): Talos 1.13.0
• Kernel Version: 6.18.24-talos
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd://2.2.3
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): K8s 1.35.5

2. Issue or feature description

We're running v26.3.3 of the GPU operator chart on Talos VMs with one H100 GPU each, attached via PCI passthrough.

The device plugin pods use 100% CPU on nodes with MIG layouts with multiple slices. On nodes where MIG is disabled or set to a single-slice layout (in our case, a single 7g.94gb partition), the CPU usage is minimal. CPU usage rises to 100% pretty much immediately after the pod starts. Known affected layouts:

• 1x 4g.47gb and 1x 3g.47gb
• 1x 3g.47gb and 2x 2g.24gb

Known unaffected layouts:

• 1x 7g.94gb
• MIG disabled, i.e. mig-enabled: false in the mig-parted configuration

The same behavior was described in #1542, but the issue was closed since OP did not respond to comments.

3. Information to attach (optional if deemed irrelevant)

From the Talos dashboard:

The logs don't show any particular errors. The pods with low and high CPU usage have basically the same output, aside from the resources advertised. The error ERROR: init 250 result=11 appears in all the NVIDIA pods but doesn't seem to affect anything. I've attached the full logs below. The pod starts the GRPC servers for the configured MIG slices and then nothing else gets printed. The device plugin responds correctly to changes to the nvidia.com/device-plugin.config label.

Note: the MIG manager doesn't work on Talos. See NVIDIA/mig-parted#382, which adds support for Talos. We are using this build.

Common error checking:

• The output of nvidia-smi -a on your host
• Your docker configuration file (e.g: /etc/docker/daemon.json)
• The k8s-device-plugin container logs
  • device-plugin.log
• The kubelet logs on the node (e.g: sudo journalctl -r -u kubelet)

Additional information that might help better understand your environment and reproduce the bug:

• Docker version from docker version
• Docker command, image and tag used
• Kernel version from uname -a
• Any relevant kernel output lines from dmesg
• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
  • Installed via Talos system extensions
  • extensions.talos.dev/nonfree-kmod-nvidia-lts=580.126.20-v1.13.0
  • extensions.talos.dev/nvidia-container-toolkit-lts=580.126.20-v1.19.0
• NVIDIA Container Toolkit version from nvidia-ctk --version

[rajathagasthya]

Thanks for detailed report and logs @Arc676! The pprof from that issue shows the CPU is spent inside NVML's nvmlEventSetWait in the health check loop. It looks like with a mixed MIG layout the plugin runs one health checker (and one NVML event set) per resource type.

Couple of questions:

1. If you set DP_DISABLE_HEALTHCHECKS=all on the device plugin, does CPU usage return to normal?
2. Does a multiple count, single profile (e.g. 2x 3g.47gb) also show high CPU?

[avasilevskii]

Data point from the mock side: we hit this exact signature (100% of one core, nothing logged after Ignoring the following XIDs...) with NVIDIA/k8s-test-infra's nvml-mock, on a plain non-MIG setup, because its nvmlEventSetWait returned ERROR_TIMEOUT immediately instead of blocking. The health loop in internal/rm/health.go has no delay of its own: eventSet.Wait(5000) blocking is the delay, so an early return turns it into a tight loop. Making the mock honor the timeout removed the CPU burn entirely (NVIDIA/k8s-test-infra#535)

That may or may not be your root cause, and the pprof probably won't distinguish it: samples land on nvmlEventSetWait both when the call returns instantly in a tight loop and when NVML spins internally during a real 5 s block. Try separating them by runningstrace -c -f -p <pid> for ~10 s: a huge ioctl count means early return.

Either way, the loop looks worth hardening: a floor so it can't iterate faster than the requested timeout, plus backoff on the ret != SUCCESS path (which currently also re-sends every device to the unhealthy channel and logs once per iteration). Happy to open a PR.

[Arc676]

@rajathagasthya Disabling the healthchecks did indeed resolve the issue, thanks! Where can I find out more about what the health checks entail? I'd like to employ this workaround for the time being, assuming the health checks aren't too critical.

I tried a couple single-profile setups as well. In all instances, the CPU usage was at normal levels. This is also an interesting alternative workaround for us, since it means that we can bypass the issue by not creating the MIG devices until we need them.

• 7x 1g.12gb
• 2x 3g.47gb
• 2x and 3x 2g.24gb (deliberately chosen as example setups that don't use the full GPU)

@avasilevskii I attached a debug container to one of the busy pods; here's some strace output:

 nvidia-device-plugin-daemonset-l9r6p  ~  ps                                                                                                                                                                 [0/383]
PID   USER     TIME  COMMAND
    1 65535     0:00 /pause
   64 root     41:50 nvidia-device-plugin
  125 root      0:00 config-manager
  189 root      0:01 zsh
  613 root      0:00 ps

 nvidia-device-plugin-daemonset-l9r6p  ~  timeout -s INT 10 strace -cfp 64 -e quiet=attach
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
100.00    0.000006           0        31         2 futex
  0.00    0.000000           0         1           poll
  0.00    0.000000           0         2           pread64
  0.00    0.000000           0        11           nanosleep
  0.00    0.000000           0         2           getpid
  0.00    0.000000           0         3           sched_yield
  0.00    0.000000           0         2           sched_getaffinity
  0.00    0.000000           0         2           restart_syscall
  0.00    0.000000           0         3           epoll_pwait
------ ----------- ----------- --------- --------- ------------------
100.00    0.000006           0        57         2 total

 nvidia-device-plugin-daemonset-l9r6p  ~  timeout -s INT 10 strace -cfp 64 -e quiet=attach
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 49.11    0.007798        3899         2           restart_syscall
 25.14    0.003992        1330         3           futex
 25.14    0.003992        3992         1           poll
  0.61    0.000097          97         1           epoll_pwait
  0.00    0.000000           0         2           pread64
  0.00    0.000000           0         2           nanosleep
  0.00    0.000000           0         2           getpid
  0.00    0.000000           0         2           sched_getaffinity
------ ----------- ----------- --------- --------- ------------------
100.00    0.015879        1058        15           total

The distribution isn't consistent. Sometimes it's 100% restart_syscall or 50/50 between futex and poll. Sometimes all the calls are at 0%. I didn't see anything regarding ioctl, though. I'm using nicolaka/netshoot for the debug container; perhaps the version of strace in the image doesn't match the one you had in mind.

 nvidia-device-plugin-daemonset-l9r6p  ~  strace --version
strace -- version 6.19
Copyright (c) 1991-2026 The strace developers <https://strace.io>.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Optional features enabled: stack-trace=libdw stack-demangle no-m32-mpers no-mx32-mpers

[rajathagasthya]

@Arc676 Thanks for the confirmation! As far as I know, health check looks for critical XID errors and marks the device unhealthy which makes Kubelet stop scheduling new pods onto it. It won't recover until XID errors are resolved and the plugin is restarted. If you disable it, it will cost you automatic fencing of faulty devices.

[Arc676]

Got it, thanks for the update. We probably won't disable the health checks in production then. The alternative workaround I mentioned looks like a good temporary solution.