Add GPU utilization Grafana dashboard with dcgm-exporter metrics

[h0pers]

What you would like to be added?

Add a Grafana dashboard for TrainJob GPU utilization using dcgm-exporter metrics. This would complement the controller health dashboard added in #3653 by providing visibility into GPU resource usage during training.

The dashboard could be modeled after the NVIDIA dcgm-exporter dashboard and adapted for TrainJob workloads.

It should follow the same grafanaDashboard selection pattern introduced in #3653, allowing users to enable it individually or via grafanaDashboard.defaultEnabled.

Why is this needed?

GPU utilization is a key metric for ML training workloads. Users running TrainJobs on GPU nodes need visibility into GPU memory usage, compute utilization, and temperature to optimize training performance and cost. Currently, the Grafana dashboard in #3653 only covers controller health and reconciliation metrics.

Ref: #3653 (comment)

[sanskar-singh-2403]

I'd be interested in picking this up. Since it builds on the grafanaDashboard selection pattern from #3653 which is still open, should this wait until that PR merges, or is the intent to stack it on top? Also, @h0pers - are you planning to handle this or else i would love to pick this up along with #3804? Thanks!

[h0pers]

Hi @sanskar-singh-2403! Sure, you can work on it. This issue depends on #3653, so we would need to wait until that PR is merged before continuing here. Feel free to pick whichever one you like the most. Thanks!

[h0pers]

/assign @sanskar-singh-2403

[sanskar-singh-2403]

Thanks @h0pers, sounds good. I will hold off until #3653 is merged so I can build on the final grafanaDashboard selection pattern, then pick this up.

Since #3804 stacks on the same base dashboard and is largely panel additions, I am happy to take both #3803 and #3804 together so the GPU/dcgm panels and the extra controller panels land consistently. I will start with #3803 once #3653 is in. Feel free to assign me #3804 as well if that works for you.

[h0pers]

Sure, you can self sign yourself with /assign command, just comment under #3804 issue. It seems I can't do it for you because you haven't commented on this issue before.