k3s: how to expose DCGM exporter metrics to Prometheus cleanly?

[iacker]

Running DCGM exporter as a DaemonSet on k3s. What is the cleanest way to get its metrics scraped: a ServiceMonitor via the Prometheus operator, or a plain scrape-config annotation?

I want per-GPU labels (uuid, model) to survive into Grafana without manual relabeling. Curious what this lab does.

[iacker]

Use a ServiceMonitor with the Prometheus operator rather than scrape annotations.

Steps that work cleanly on k3s:

1. Run DCGM exporter as a DaemonSet with a headless Service exposing port 9400 (name it metrics).
2. Create a ServiceMonitor selecting that Service; set honorLabels: true so exporter-provided labels win.
3. DCGM already emits gpu, UUID, modelName, Hostname labels per metric — with honorLabels they flow straight into Prometheus, no manual relabeling.
4. In Grafana, template a $gpu variable on label_values(DCGM_FI_DEV_GPU_UTIL, gpu) and key panels off UUID.

Annotation-based scraping also works but you lose the operators native target management and have to relabel by hand, so the ServiceMonitor path is the one this lab standardizes on.