diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index 032ae0263..cb01ae13d 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -20,256 +20,387 @@ DRA Driver for NVIDIA GPUs ########################## -Dynamic Resource Allocation (DRA) is a Kubernetes concept for flexibly requesting, configuring, and sharing specialized devices like GPUs. -This page describes how to install and upgrade the DRA Driver for NVIDIA GPUs v${dra_version} with the NVIDIA GPU Operator. +Dynamic Resource Allocation (DRA) is a Kubernetes API for flexibly requesting, configuring, and sharing specialized devices such as GPUs. +This page describes how to use the GPU Operator to install and manage DRA Driver for NVIDIA GPUs v${dra_version}. -Before using the DRA Driver for NVIDIA GPUs, it is recommended that you are familiar with the following: +Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the following documentation: -* `Upstream Kubernetes DRA documentation `_. -* `DRA Driver documentation `__. +* `Dynamic Resource Allocation `_ in the Kubernetes documentation. +* `DRA Driver for NVIDIA GPUs documentation `__. -***************** +********************************* +Comparison: DRA and Device Plugin +********************************* + +GPU Operator supports the DRA Driver as a more recent alternative to the NVIDIA Device Plugin for Kubernetes. +However, some features of the DRA driver have alpha support status. + +A cluster can have either a ``GPUCluster`` resource for DRA or a ``ClusterPolicy`` resource for the Device Plugin, but not both. + +Capability Comparison +===================== + +The following table compares the GPU allocation capabilities of the two mechanisms when managed by the GPU Operator: + +.. list-table:: GPU Allocation Capability Comparison + :header-rows: 1 + :widths: 23 38 39 + + * - Capability + - DRA with ``GPUCluster`` + - Device Plugin with ``ClusterPolicy`` + + * - API and Device Selection + - Uses DRA API to select GPUs by attributes---model, architecture, memory, compute capability, UUID, addressing mode, PCI/topology information---using CEL expressions. + + With Kubernetes v1.36, the driver also supports the extended resource names for backward compatibility. + - Uses Kubernetes extended resource names and counts. + Node labeling or other external mechanisms required for structured selection. + + * - Full GPU and MIG Device Allocation + - Allocates full GPU devices and preconfigured MIG devices. + Alpha ``DynamicMIG`` creates MIG devices for a claim. + - Allocates full GPU devices or preconfigured MIG devices. + MIG Manager configures MIG geometry before allocation. + + * - GPU Sharing + - Alpha ``ConsumableShares`` feature gate implements Kubernetes consumable capacity for scheduler-accounted sharing among independent ``ResourceClaim`` objects for full GPUs and MIG devices. + The driver also supports time-slicing through a shared DRA claim. + Custom time-slice intervals require the Alpha ``TimeSlicingSettings`` feature gate. + MPS requires the Alpha ``MPSSupport`` gate. + - Advertises time-slicing replicas and experimental MPS replicas for all GPUs on a node. + + * - Multi-Node NVLink + - Provides the ComputeDomain API and controller for coordinated workloads. + - Does not provide the ComputeDomain API. + + * - Device Health Reporting + - Alpha ``NVMLDeviceHealthCheck`` is disabled by default. + gRPC health probes report kubelet plugin availability only. + - Reports unhealthy devices through the Kubernetes Device Plugin API. + + * - Device Injection and Managed Components + - Requires a Container Device Interface (CDI)-compatible runtime. + ``GPUCluster`` manages DRA, ComputeDomains, NVIDIA Data Center GPU Manager (DCGM), DCGM Exporter, and the DRA validator. + - Supports multiple device injection strategies. + ``ClusterPolicy`` manages NVIDIA Container Toolkit, GPU Feature Discovery, MIG Manager, the sandbox device plugin, Kata Containers, KubeVirt, and NVIDIA vGPU Manager. + + * - Kubernetes Scheduling + - Allocation through a ``ResourceClaim``, which can be referenced by multiple containers or Pods. + - Allocation through the resource limits specified for the container. + +DRA Driver Feature Maturity +=========================== + +The following table groups the DRA driver capabilities by maturity: + +.. list-table:: DRA Driver Capability Maturity + :header-rows: 1 + :widths: 34 22 18 26 + + * - Capability + - Support Status + - Default + - Operator Control + + * - ComputeDomains + - Generally Available + - Enabled + - ``draDriver.computeDomains.enabled`` + + * - Full GPU and existing MIG device allocation + - Generally Available + - Enabled + - Always enabled + + * - ``ComputeDomainCliques``, ``CrashOnNVLinkFabricErrors``, ``IMEXDaemonsWithDNSNames`` + - Generally Available + - Enabled + - ``draDriver.featureGates`` + + * - ``ConsumableShares``, ``DeviceMetadata``, ``DynamicMIG``, ``MPSSupport``, ``NVMLDeviceHealthCheck``, ``PassthroughSupport``, ``TimeSlicingSettings`` + - Alpha + - Disabled + - ``draDriver.featureGates`` + +Refer to the `DRA driver v${dra_version} overview `__ +and `feature-gate definitions `__ +for the authoritative support status, maturity, and default values. + +DRA Driver Limitations +====================== + +Consider the following limitations before selecting DRA driver v${dra_version}: + +* ``NVMLDeviceHealthCheck`` is alpha and disabled by default. + The gRPC health probe reports kubelet plugin availability, and DCGM provides telemetry. + Neither provides DRA allocation health status. +* Consumable capacity is available through the alpha ``ConsumableShares`` feature gate, which is disabled by default. + When enabled, the feature provides scheduler-accounted capacity sharing among independent ``ResourceClaim`` objects for full GPUs and MIG devices. + Kubernetes v1.34 and v1.35 require enabling the ``DRAConsumableCapacity`` feature gate. + Kubernetes v1.36 and later enable the feature by default. + You must also set the ``CONSUMABLE_SHARES`` environment variable for the DRA GPU kubelet plugin to ``memory``, ``unlimited``, or a positive integer. + Set the environment variable through ``draDriver.gpus.kubeletPlugin.env``. + MPS cannot be used when consumable capacity is configured through ``ConsumableShares``. +* Alpha feature gates have compatibility constraints. + ``DynamicMIG`` conflicts with ``PassthroughSupport``, ``NVMLDeviceHealthCheck``, and ``MPSSupport``. + ``PassthroughSupport`` conflicts with ``NVMLDeviceHealthCheck``, and ``DeviceMetadata`` requires ``PassthroughSupport``. +* Kubernetes does not support preemption for DRA resources. + Injecting DRA-allocated devices requires a preconfigured CDI-compatible container runtime. + The managed ``gpu.nvidia.com`` ``DeviceClass`` does not set ``spec.extendedResourceName`` to ``nvidia.com/gpu``. + +******** Overview -***************** +******** -With the DRA Driver for NVIDIA GPUs, your Kubernetes workloads can allocate and consume the following two types of resources: +The GPU Operator manages DRA components through the ``GPUCluster`` custom resource. +``GPUCluster`` is a cluster-scoped singleton and its name must be ``gpu-cluster``. +The Helm chart creates this resource when you set ``gpuCluster.deployCR=true``. -* GPU allocation: for controlled sharing and dynamic reconfiguration of GPUs. - This functionality replaces the traditional GPU allocation method used by the NVIDIA Kubernetes Device Plugin. -* ComputeDomains: an abstraction for secure `Multi-Node NVLink (MNNVL) `_ for NVIDIA GB200 and similar systems. +A cluster can have either a ``GPUCluster`` or a ``ClusterPolicy`` resource, but not both. +The GPU Operator uses ``ClusterPolicy`` to manage components for Device Plugin-based allocation +and ``GPUCluster`` to manage components for DRA-based allocation. -You can use these features independently or together in the same cluster. +For ``GPUCluster``, the GPU Operator manages the following components: -.. _known-issues: +* The DRA driver GPU capability for ``gpu.nvidia.com``, ``mig.nvidia.com``, and ``vfio.gpu.nvidia.com`` devices. + This capability is always enabled, although VFIO passthrough is unusable without the Alpha ``PassthroughSupport`` feature gate which is off by default. +* The ComputeDomain controller and kubelet plugin for Multi-Node NVLink (MNNVL). + ComputeDomain support is enabled by default and can be disabled. +* A DRA validator that allocates a GPU by using a ``ResourceClaim`` and verifies that the device is usable. +* DCGM Exporter, which is enabled by default. +* Standalone DCGM, which is disabled by default. -Known Issues -************ +The ``GPUCluster`` resource does not manage the NVIDIA GPU driver. +Use an ``NVIDIADriver`` resource to install a containerized driver or use a driver that is pre-installed on the host. +The Operator automatically assigns the ``nvidia.com/gpu.deploy.*`` labels that control operand placement. -This section covers known issues for the DRA Driver when used with the NVIDIA GPU Operator. -For known issues specific to the DRA Driver itself, refer to the `DRA Driver v${dra_version} release notes `__. +GPUCluster Limitations +====================== -* There is a known issue where the NVIDIA Driver Manager is not aware of the DRA driver kubelet plugin, and will not correctly evict it on pod restarts. - You must label the nodes you plan to use with DRA GPU allocation and pass the node label in the GPU Operator Helm command in the ``driver.manager.env`` flag. - This enables the NVIDIA Driver Manager to evict the GPU kubelet plugin correctly on driver container upgrades. -* For A100 GPUs, the MIG manager does not automatically evict the DRA kubelet plugin during MIG configuration changes. - If the DRA kubelet plugin is deployed before a MIG change, then you must manually restart the DRA kubelet plugin. +* Use this workflow for a new installation. + An in-place migration from a standalone DRA driver Helm release or from ``ClusterPolicy`` to ``GPUCluster`` is not supported. +* ``GPUCluster`` does not expose the controller affinity, priority class, toleration, + and kubelet-plugin node selector overrides that were used by the previous standalone DRA driver procedure for Google Kubernetes Engine (GKE). + This page does not provide a managed DRA installation procedure for GKE. ************* Prerequisites ************* +In addition to ensuring that your GPUs and cluster align with the :ref:`GPU Operator support matrix `, verify the following prerequisites: -In addition to ensuring your GPUs and cluster align with the :ref:`GPU Operator support matrix `, the following prerequisites must be met: +* Kubernetes v1.34.2 or later with a ``resource.k8s.io`` ``DeviceClass`` API available. +* NVIDIA GPU driver version 580 or later. +* An underlying container runtime that supports CDI and is configured to use CDI for injecting DRA-allocated devices. +* No ``ClusterPolicy`` resource exists in the cluster. -* Kubernetes v1.34.2 or later. + .. code-block:: console - .. note:: - If you plan to use traditional extended resource requests such as ``nvidia.com/gpu`` alongside the DRA driver, you must enable the `DRAExtendedResource `_ feature gate. - This enabled by default in Kubernetes v1.36.0 and later. - It allows the scheduler to translate extended resource requests into ResourceClaims for the DRA driver. - -* For GPU allocation support, you must label nodes that will support GPU allocation with ``nvidia.com/dra-kubelet-plugin=true`` and use this label as a node selector in the DRA driver Helm chart. - This is required to avoid the :ref:`known issue ` when using the GPU Operator with the DRA Driver. - Steps for labeling nodes are provided in the install section. - The label is also passed to the GPU Operator Helm command via the ``driver.manager.env`` flag. - -.. _computedomain-prereqs: + $ kubectl get clusterpolicy -* For ComputeDomain ensure the following: + If a ``ClusterPolicy`` exists, use a new cluster for the ``GPUCluster`` workflow. - * NVIDIA Grace Blackwell GPUs with Multi-Node NVLink (MNNVL) available on your cluster. - For example, NVIDIA HGX GB200 NVL72 or NVIDIA HGX GB300 NVL72. - Refer to the `NVIDIA Multi-Node NVLink Systems documentation `_ for details on Multi-Node NVLink systems. - - * For using ComputeDomains with a pre-installed GPU Driver: - - * The corresponding nvidia-imex-* packages installed through your Linux distribution's package manager. - * The IMEX systemd service disabled before installing the GPU Operator (on all GPU nodes). - For example: - - .. code-block:: console +.. note:: - $ systemctl disable --now nvidia-imex.service && systemctl mask nvidia-imex.service + To use an extended-resource request with the DRA driver, enable the + `DRAExtendedResource `_ feature gate. + This feature gate is enabled by default in Kubernetes v1.36.0 and later. + Request ``deviceclass.resource.kubernetes.io/gpu.nvidia.com`` to use the managed ``gpu.nvidia.com`` ``DeviceClass``. + The legacy name ``nvidia.com/gpu`` requires a separate ``DeviceClass`` that sets ``spec.extendedResourceName`` to that value. +.. _computedomain-prereqs: -.. note:: +For ComputeDomain support, verify the following additional prerequisites: - Installing GPU Operator ${version} configures some additional DRA Driver for NVIDIA GPUs prerequisites for you: +* NVIDIA Grace Blackwell GPUs with Multi-Node NVLink available on your cluster, + such as NVIDIA HGX GB200 NVL72 or NVIDIA HGX GB300 NVL72. + Refer to the `NVIDIA Multi-Node NVLink Systems documentation `_ for more information. +* If you use a pre-installed GPU driver, install the corresponding ``nvidia-imex-*`` packages through the Linux distribution package manager. +* If you use a pre-installed GPU driver, disable and mask the IMEX systemd service on every GPU node before installing the GPU Operator: - * Container Device Interface (CDI) enabled in the underlying container runtime (such as containerd or CRI-O) - * NVIDIA Driver version 580 or later. - * Deploy Node Feature Discovery (NFD) and GPU Feature Discovery (GFD) on your cluster. + .. code-block:: console + $ systemctl disable --now nvidia-imex.service + $ systemctl mask nvidia-imex.service ******* Install ******* -This section covers fresh installs of the GPU Operator and DRA Driver for NVIDIA GPUs. -If you are upgrading an earlier version of the DRA Driver for NVIDIA GPUs, refer to the :ref:`Upgrade ` section. - -.. note:: - The ``nvidiaDriverRoot`` flag sets the root directory for the NVIDIA GPU driver. - The default value is ``/``, which is typical for drivers installed directly on the host. - With GPU Operator–managed drivers (default), drivers are installed to ``/run/nvidia/driver``. - If you are using `pre-installed drivers `_, remove the ``nvidiaDriverRoot`` flag or set it to ``/``. - +The following procedures install the GPU Operator and create the ``gpu-cluster`` resource. +The Operator deploys the DRA driver operands in the GPU Operator namespace. -.. tab-set:: - :sync-group: dra +#. Add the NVIDIA Helm repository: - .. tab-item:: GPU Allocation - :sync: gpu-allocation + .. code-block:: console - 1. Label every node that will support GPU allocation through DRA: + $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ + && helm repo update - .. code-block:: console +#. Install the GPU Operator by using the procedure for your driver configuration. - $ kubectl label node $HOSTNAME nvidia.com/dra-kubelet-plugin=true + .. tab-set:: + :sync-group: dra-driver-install - 2. Add the NVIDIA Helm repository: + .. tab-item:: Operator-Managed Driver + :sync: managed .. code-block:: console - $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ - && helm repo update + $ helm upgrade --install gpu-operator nvidia/gpu-operator \ + --version=${version} \ + --namespace gpu-operator \ + --create-namespace \ + --set clusterPolicy.deployCR=false \ + --set gpuCluster.deployCR=true \ + --set driver.nvidiaDriverCRD.enabled=true + + The chart creates the default ``NVIDIADriver`` resource to manage the NVIDIA GPU driver. - 3. Install the GPU Operator with the NVIDIA Kubernetes Device Plugin disabled: + .. tab-item:: Pre-Installed Driver + :sync: preinstalled .. code-block:: console $ helm upgrade --install gpu-operator nvidia/gpu-operator \ --version=${version} \ - --create-namespace \ --namespace gpu-operator \ - --set devicePlugin.enabled=false \ - --set driver.manager.env[0].name=NODE_LABEL_FOR_GPU_POD_EVICTION \ - --set driver.manager.env[0].value="nvidia.com/dra-kubelet-plugin" + --create-namespace \ + --set clusterPolicy.deployCR=false \ + --set gpuCluster.deployCR=true \ + --set driver.enabled=false - Make sure the value of ``driver.manager.env`` matches the node label applied in step 1. +By default, the Operator enables both GPU allocation and ComputeDomain support. +To install GPU allocation without ComputeDomain support, add the following option to the Helm command: - Make sure the ``devicePlugin.enabled`` flag is set to ``false`` to disable the NVIDIA Kubernetes Device Plugin. - The DRA Driver for NVIDIA GPUs will be used to allocate GPUs. +.. code-block:: console - Refer to the `GPU Operator installation guide `_ for additional configuration options. - If you plan to use MIG devices, refer to the `GPU Operator MIG documentation `_ to configure your cluster for MIG support. - Restart the DRA driver pod to discover MIG devices created by the GPU Operator MIG Manager. + --set draDriver.computeDomains.enabled=false - 4. Create a ``values.yaml`` file for the DRA driver: +Do not install a separate ``dra-driver-nvidia-gpu`` Helm release for this managed workflow. - .. code-block:: yaml +************************ +Configure DRA Components +************************ - image: - pullPolicy: IfNotPresent - kubeletPlugin: - nodeSelector: - nvidia.com/dra-kubelet-plugin: "true" +The GPU Operator Helm values render the specification of the ``gpu-cluster`` resource. +The following settings provide the primary configuration surface: - If you are using Google Kubernetes Engine (GKE), the DRA driver requires additional overrides for the driver root, controller affinity, and tolerations: +.. list-table:: + :header-rows: 1 + :widths: 38 62 - .. code-block:: yaml + * - Helm value + - Description - # GKE helm values example - # "/home/kubernetes/bin/nvidia" is the default driver root on GKE. - nvidiaDriverRoot: "/home/kubernetes/bin/nvidia" + * - ``draDriver.repository``, ``draDriver.image``, and ``draDriver.version`` + - Configure the DRA driver container image. - controller: - priorityClassName: "" - affinity: null - image: - pullPolicy: IfNotPresent - kubeletPlugin: - priorityClassName: "" - tolerations: - - effect: NoSchedule - key: nvidia.com/gpu - operator: Exists - nodeSelector: - nvidia.com/dra-kubelet-plugin: "true" + * - ``draDriver.imagePullPolicy`` and ``draDriver.imagePullSecrets`` + - Configure image pulling for the DRA driver containers. - 5. Install the DRA driver: + * - ``draDriver.featureGates`` + - Enable or disable DRA driver feature gates. + The Operator renders the map as the ``FEATURE_GATES`` environment variable for DRA driver containers. - .. code-block:: console + * - ``draDriver.gpus.kubeletPlugin`` + - Configure environment variables, resource requests and limits, and the gRPC health check for the GPU kubelet-plugin container. - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set nvidiaDriverRoot=/run/nvidia/driver \ - --set gpuResourcesEnabledOverride=true \ - -f values.yaml + * - ``draDriver.computeDomains.enabled`` + - Enable or disable the ComputeDomain controller and kubelet-plugin container. - For GKE, omit ``--set nvidiaDriverRoot=/run/nvidia/driver``; the value comes from the GKE ``values.yaml`` file. + * - ``draDriver.computeDomains.controller`` + - Configure environment variables and resource requests and limits for the ComputeDomain controller. - .. tab-item:: ComputeDomain - :sync: computedomain + * - ``draDriver.computeDomains.kubeletPlugin`` + - Configure environment variables, resource requests and limits, and the gRPC health check for the ComputeDomain kubelet-plugin container. - 1. Add the NVIDIA Helm repository: + * - ``hostPaths.kubeletRootDir`` + - Configure the kubelet root directory when it differs from ``/var/lib/kubelet``. - .. code-block:: console + * - ``daemonsets`` + - Configure common labels, annotations, and tolerations. + ``daemonsets.priorityClassName`` applies to the DRA driver kubelet plugin and ComputeDomain controller. - $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update +For example, the following values enable a DRA driver feature gate, configure container resources, +and enable the default gRPC health checks on their default ports: - 2. Install the GPU Operator: +.. code-block:: yaml - .. code-block:: console + draDriver: + featureGates: + NVMLDeviceHealthCheck: true + gpus: + kubeletPlugin: + resources: + requests: + cpu: 50m + memory: 64Mi + healthcheck: + enabled: true + computeDomains: + enabled: true + controller: + resources: + requests: + cpu: 50m + memory: 64Mi + kubeletPlugin: + healthcheck: + enabled: true - $ helm upgrade --install gpu-operator nvidia/gpu-operator \ - --version=${version} \ - --create-namespace \ - --namespace gpu-operator +The GPU kubelet-plugin health-check port defaults to ``51516`` and the ComputeDomain kubelet-plugin health-check port defaults to ``51515``. +The Operator also enables ComputeDomain GPU clique labeling by setting ``GPU_CLIQUE_LABEL_ENABLED=true`` automatically. - Refer to the `GPU Operator installation guide `_ for additional configuration options. - - 3. Install the DRA driver. +********************* +Validate Installation +********************* - Example for a GPU Operator–managed driver: +Reconciliation typically completes within 3 minutes. +During reconciliation, the STATUS column progresses from empty to `notReady` to `ready`. - .. code-block:: console +#. Verify that the ``GPUCluster`` resource is ready: - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set resources.gpus.enabled=false \ - --set nvidiaDriverRoot=/run/nvidia/driver + .. code-block:: console - Example for a pre-installed GPU driver: + $ kubectl get gpucluster gpu-cluster - .. code-block:: console + *Example Output* - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set resources.gpus.enabled=false + .. code-block:: output -********************* -Validate Installation -********************* + NAME STATUS AGE + gpu-cluster ready 3m -1. Confirm that the DRA driver components are running: + If the resource does not become ready, inspect its conditions and recent events: .. code-block:: console - $ kubectl get pods -n nvidia-dra-driver-gpu + $ kubectl describe gpucluster gpu-cluster - *Example Output* + A ``PrerequisiteNotMet`` condition can indicate that a ``ClusterPolicy`` resource exists. + An ``OperandNotReady`` condition indicates that the Operator is waiting for one or more managed pods. - .. code-block:: output +#. Confirm that the managed components are running in the GPU Operator namespace: - NAME READY STATUS RESTARTS AGE - dra-driver-nvidia-gpu-controller-67cb99d84b-5q7kj 1/1 Running 0 7m26s - dra-driver-nvidia-gpu-kubelet-plugin-h5xsn 2/2 Running 0 7m27s + .. code-block:: console + + $ kubectl get pods -n gpu-operator - The controller pod runs the ComputeDomain controller (1 container). The kubelet-plugin pod runs two containers, one for GPU resources (gpus) and one for ComputeDomain resources (compute-domains), so it shows ``2/2`` when both are enabled. One kubelet-plugin pod appears per GPU node. + Expected workload names include the following: - If you installed with ``--set resources.computeDomains.enabled=false``, the controller pod is not present and the kubelet-plugin pod shows ``1/1``. The same is true if you disabled GPU allocation during install. + * ``nvidia-dra-driver-kubelet-plugin`` + * ``nvidia-dra-driver-controller`` when ComputeDomain support is enabled + * ``nvidia-dra-validator`` + * ``nvidia-dcgm-exporter-dra`` when DCGM Exporter is enabled + * ``nvidia-dcgm-dra`` when standalone DCGM is enabled - .. note:: - If you upgraded an existing v25.x installation, the pod names retain the ``nvidia-dra-driver-gpu-`` prefix (for example, ``nvidia-dra-driver-gpu-controller-*``) because the upgrade preserves the original resource names through the ``nameOverride`` flag. + The DRA kubelet-plugin DaemonSet runs a GPU container and, + when ComputeDomain support is enabled, a ComputeDomain container. + The DRA validator and enabled telemetry operands run one pod on each GPU node. -2. Verify that GPU DeviceClasses are available: +#. Verify that the DeviceClasses are available: .. code-block:: console @@ -279,62 +410,163 @@ Validate Installation .. code-block:: output - NAME AGE - compute-domain-daemon.nvidia.com 55s - compute-domain-default-channel.nvidia.com 55s - gpu.nvidia.com 55s - mig.nvidia.com 55s + compute-domain-daemon.nvidia.com 2m + compute-domain-default-channel.nvidia.com 2m + gpu.nvidia.com 2m + mig.nvidia.com 2m + vfio.gpu.nvidia.com 2m + + The ComputeDomain DeviceClasses are present only when ComputeDomain support is enabled. + +************************* +Running a Sample Workload +************************* -The ``compute-domain-daemon.nvidia.com`` and ``compute-domain-default-channel.nvidia.com`` DeviceClasses are installed when ComputeDomain support is enabled. -The ``gpu.nvidia.com`` and ``mig.nvidia.com`` DeviceClasses are installed when GPU allocation support is enabled. +Refer to `Request full GPUs `__ +in the DRA driver documentation for information about the following workload-related tasks: -Additional validation steps are available in the upstream DRA Driver documentation: +* Request any GPU +* Request multiple GPUs in one pod +* Share a GPU across containers in a pod +* Select a GPU by product name +* Select a GPU by memory size +* Combine attribute and capacity selectors -* `Validate setup for ComputeDomain allocation `_ -* `Validate setup for GPU allocation `_ +********* +Telemetry +********* +DCGM Exporter is enabled by default with ``GPUCluster`` and uses its embedded host engine. +Set ``dcgm.enabled=true`` to deploy the standalone ``nvidia-dcgm-dra`` host engine instead. + +When either ``dcgmExporter.enablePodLabels`` or ``dcgmExporter.enablePodUID`` is enabled, +the Operator enables DRA ``ResourceSlice`` attribution in DCGM Exporter and grants the exporter read access to ``ResourceSlice`` objects. +This enables pod metadata enrichment for GPUs allocated through DRA. + +.. _dra-driver-upgrade: .. _upgrade: ******* Upgrade ******* -Starting with v0.4.0, the DRA Driver for NVIDIA GPUs moved to ``kubernetes-sigs/dra-driver-nvidia-gpu`` and adopted semantic versioning. -The Helm chart was renamed from ``nvidia-dra-driver-gpu`` to ``dra-driver-nvidia-gpu`` and is published to new NGC Helm and container registries. +The DRA driver is upgraded as part of the GPU Operator release. +When you upgrade the GPU Operator, preserve ``clusterPolicy.deployCR=false`` and ``gpuCluster.deployCR=true`` in your values. +You can select a different DRA driver version by setting ``draDriver.version``. +Refer to :ref:`Upgrading the NVIDIA GPU Operator` for the Operator upgrade procedure and required CRD updates. + +During an Operator-managed NVIDIA GPU driver upgrade, pods with allocated ``gpu.nvidia.com`` ResourceClaims are treated as GPU workloads. +The driver upgrade policy applies to those pods, and the DRA kubelet plugin remains available while claims are unprepared. +You do not need to configure a separate node label for DRA workload eviction. + +This procedure does not migrate a standalone DRA driver Helm release to ``GPUCluster``. +For a standalone installation, refer to the `upstream DRA driver upgrade guide `__ +or select the GPU Operator documentation version that matches the installed release. + +********* +Uninstall +********* + +Before uninstalling the GPU Operator, delete user workloads and user-created ResourceClaims for GPUs +and verify that the workload pods terminate and the claims are unprepared. +Afterward, uninstall the GPU Operator by using Helm. + +The chart includes a ``pre-delete`` hook that deletes ``gpu-cluster`` and waits for the GPUCluster finalizer to perform ordered teardown. + +.. tip:: The uninstall command output can report that the ``gpu-cluster`` resource was kept. + + However, a pre-delete hook actually deletes the resource. + + .. code-block:: console + + These resources were kept due to the resource policy: + [GPUCluster] gpu-cluster + + release "gpu-operator" uninstalled + +The finalizer removes Operator-managed DaemonSets that consume ResourceClaims before the DRA kubelet plugin is removed. +Do not use ``helm uninstall --no-hooks`` while ``gpu-cluster`` exists because Helm can remove the Operator before this teardown completes. + +Refer to :doc:`uninstall` for the complete uninstall procedure and CRD cleanup information. + +*************** +Troubleshooting +*************** -When upgrading from v25.x you must explicitly set ``nameOverride`` and ``--version`` to avoid creating duplicate Kubernetes manifests under different names. -Without ``--set nameOverride=nvidia-dra-driver-gpu``, the upgrade creates new daemonsets and deployments under the new chart name instead of upgrading the existing resources in place. +Recover From an Uninstall That Skipped the Pre-Delete Hook +========================================================== -.. important:: - After upgrading to v0.4.0, downgrading to v25.x is not supported. +If you run ``helm uninstall gpu-operator --no-hooks`` while ``gpu-cluster`` exists, +Helm removes the Operator before the ``pre-delete`` hook can perform an ordered teardown. +The following resources are left in the cluster with no controller to reconcile them: -Upgrade from v25.x to v0.4.0 or later -************************************** +* The ``gpu-cluster`` resource, which cannot be deleted because the + ``gpucluster.nvidia.com/dra-resourceclaim`` finalizer requires the Operator. +* The DRA driver kubelet plugin, DRA validator, and DCGM Exporter pods. +* Operator-managed ResourceClaims for the preceding pods. -1. Apply the v${dra_version} CRDs for ComputeDomains and ComputeDomainsCliques before upgrading the Helm chart. - Refer to the `v${dra_version} release page `__ for the CRD manifests. +To recover, remove the finalizer from ``gpu-cluster``, then force-delete the orphaned operands: -2. Run ``helm upgrade`` with ``nameOverride`` and ``--version``, preserving any original install flags such as ``gpuResourcesEnabledOverride`` and ``nvidiaDriverRoot``: +#. Remove the finalizer from ``gpu-cluster``: .. code-block:: console - $ helm upgrade -i nvidia-dra-driver-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --set nameOverride=nvidia-dra-driver-gpu \ - --set gpuResourcesEnabledOverride=true \ - --set nvidiaDriverRoot=/run/nvidia/driver + $ kubectl patch gpucluster gpu-cluster --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' -3. Verify the upgrade: +#. Remove finalizers from the orphaned ResourceClaims in the GPU Operator namespace: .. code-block:: console - $ kubectl get pods -n nvidia-dra-driver-gpu + $ for rc in $(kubectl get resourceclaim -n gpu-operator -o name); do + kubectl patch $rc -n gpu-operator --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' + done + +#. Force-delete any pods that remain in the ``Terminating`` state: + + .. code-block:: console + + $ kubectl delete pods -n gpu-operator --all --grace-period=0 --force + +#. Delete the GPU Operator namespace to remove residual state: + + .. code-block:: console + + $ kubectl delete namespace gpu-operator + +To avoid this recovery path, run ``helm uninstall`` without ``--no-hooks`` so that the +``pre-delete`` hook can complete ordered teardown. + +Clean Up ResourceClaims Stuck in ``deleted,allocated,reserved`` +=============================================================== + +ResourceClaims in the ``deleted,allocated,reserved`` state indicate that the API server +accepted a delete request, but the DRA driver kubelet plugin has not completed the +device _unprepare_ step for the claim. +This state can persist if the DRA driver kubelet plugin was removed or restarted +before it could unprepare the allocated devices. + +Take one of the following actions: + +* If the DRA driver kubelet plugin can be restarted, ensure that the + ``nvidia-dra-driver-kubelet-plugin`` pod is running. + The plugin completes the unprepare step and the ResourceClaims are removed. + +* If the DRA driver kubelet plugin cannot be restarted, such as after an + uninstall that skipped the pre-delete hook, remove the finalizers from + the affected ResourceClaims: + + .. code-block:: console - All controller and kubelet-plugin pods should reach ``Running`` status, and existing ResourceClaims should remain in the ``allocated,reserved`` state. + $ for rc in $(kubectl get resourceclaim -n gpu-operator -o name); do + kubectl patch $rc -n gpu-operator --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' + done -Refer to the `upstream upgrade guide `__ for additional detail. + Removing finalizers bypasses the unprepare step. + Use this option only during recovery when no DRA workloads are running. ************************ Additional Documentation @@ -342,5 +574,5 @@ Additional Documentation For more details on the DRA Driver for NVIDIA GPUs, refer to the following resources: -* `DRA Driver for NVIDIA GPUs documentation `__ -* `DRA Driver v${dra_version} release notes `__ \ No newline at end of file +* `DRA Driver for NVIDIA GPUs documentation `__. +* `DRA Driver v${dra_version} release notes `__. diff --git a/gpu-operator/getting-started.rst b/gpu-operator/getting-started.rst index 1f26676a1..538348bcb 100644 --- a/gpu-operator/getting-started.rst +++ b/gpu-operator/getting-started.rst @@ -362,6 +362,11 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, the Operator deploys NVIDIA Confidential Computing Manager for Kubernetes. - ``false`` + * - ``clusterPolicy.deployCR`` + - When set to ``true``, the chart creates the ``ClusterPolicy`` resource for the NVIDIA Kubernetes Device Plugin enablement stack. + Set this value to ``false`` when using ``gpuCluster.deployCR=true``. + - ``true`` + * - ``cdi.enabled`` - When set to ``true`` (default), the Container Device Interface (CDI) will be used for injecting GPUs into workload containers. @@ -416,6 +421,7 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, Kubernetes pod labels are added as Prometheus label dimensions on the GPU metrics. Enabling this option causes the Operator to provision a cluster-scoped ClusterRole and ClusterRoleBinding (``nvidia-dcgm-exporter-read-pods``) that grants the DCGM Exporter service account ``get``, ``list``, and ``watch`` access to pods. + For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects. Use ``dcgmExporter.podLabelAllowlistRegex`` to limit which labels are emitted. - ``false`` @@ -423,6 +429,7 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, the Kubernetes pod UID is added as a Prometheus label dimension on the GPU metrics. Like ``dcgmExporter.enablePodLabels``, this provisions a cluster-scoped ClusterRole and ClusterRoleBinding that grants the DCGM Exporter service account ``get``, ``list``, and ``watch`` access to pods. + For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects. - ``false`` * - ``dcgmExporter.podLabelAllowlistRegex`` @@ -471,6 +478,20 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. custom driver images. - ``nvcr.io/nvidia`` + * - ``draDriver.computeDomains.enabled`` + - When set to ``true``, the ``GPUCluster`` stack deploys the ComputeDomain controller and kubelet plugin. + Refer to :doc:`DRA Driver for NVIDIA GPUs ` for more information. + - ``true`` + + * - ``draDriver.version`` + - Version of the DRA Driver for NVIDIA GPUs deployed by the ``GPUCluster`` stack. + - ``v${dra_version}`` + + * - ``gpuCluster.deployCR`` + - When set to ``true``, the chart creates the ``GPUCluster`` resource for the DRA enablement stack. + Set ``clusterPolicy.deployCR=false`` and refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete installation procedure. + - ``false`` + * - ``driver.rdma.enabled`` - Controls whether the driver daemon set builds and loads the legacy ``nvidia-peermem`` kernel module. @@ -877,6 +898,7 @@ After verifying the installation, you can configure the GPU Operator for your wo - :doc:`gpu-sharing` — Share a single GPU across multiple pods using time-slicing or MPS. - :doc:`gpu-operator-mig` — Configure Multi-Instance GPU (MIG) partitioning on supported GPUs. - :doc:`gpu-operator-rdma` — Enable GPUDirect RDMA for high-performance networking. +- :doc:`dra-intro-install` — Allocate GPUs by using Kubernetes Dynamic Resource Allocation (DRA). - :doc:`gpu-driver-configuration` — Use the NVIDIA GPU Driver Custom Resource Definition to manage drivers per node. - :doc:`precompiled-drivers` — Speed up driver deployments with precompiled kernel modules. - :doc:`cdi` — Learn about Container Device Interface (CDI) and NRI Plugin mode. diff --git a/gpu-operator/gpu-driver-configuration.rst b/gpu-operator/gpu-driver-configuration.rst index 7fd8b68f4..b799732f3 100644 --- a/gpu-operator/gpu-driver-configuration.rst +++ b/gpu-operator/gpu-driver-configuration.rst @@ -30,6 +30,10 @@ You can create one or more instances of an NVIDIA driver (``NVIDIADriver``) cust to specify the NVIDIA GPU driver type and driver version to configure on specific nodes. You can specify labels in the node selector field to control which NVIDIA driver configuration is applied to specific nodes. +The ``GPUCluster`` enablement stack uses ``NVIDIADriver`` resources for an Operator-managed driver. +``GPUCluster`` manages the DRA operands and waits for the matching NVIDIA driver deployment to become ready. +Refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete ``GPUCluster`` installation procedure. + Driver Management Modes ======================= @@ -406,6 +410,12 @@ Installing the NVIDIA GPU Operator Perform the following steps to install the GPU Operator and use the NVIDIA driver custom resources. +.. note:: + + To use an ``NVIDIADriver`` resource with the ``GPUCluster`` DRA stack, + follow the installation procedure in :doc:`DRA Driver for NVIDIA GPUs `. + That procedure configures the chart to create ``GPUCluster`` instead of ``ClusterPolicy``. + #. Optional: If you want to run more than one driver type or version in the cluster, label the worker nodes to identify the driver type and version to install on each node: diff --git a/gpu-operator/gpu-driver-upgrades.rst b/gpu-operator/gpu-driver-upgrades.rst index 7185b7c60..1a1e98e45 100644 --- a/gpu-operator/gpu-driver-upgrades.rst +++ b/gpu-operator/gpu-driver-upgrades.rst @@ -46,6 +46,10 @@ The GPU Operator supports several methods for managing and automating this drive The GPU Operator only manages the lifecycle of containerized drivers. Drivers which are pre-installed on the host are not managed by the GPU Operator. +For the ``GPUCluster`` DRA stack, the Operator treats pods with an allocated +``gpu.nvidia.com`` ResourceClaim as GPU workloads during a driver upgrade. +The configured pod deletion policy applies to these workloads before the driver is reloaded. +The DRA kubelet plugin remains available while terminating pods unprepare their claims. ************************************ Upgrades with the Upgrade Controller diff --git a/gpu-operator/install-gpu-operator-air-gapped.rst b/gpu-operator/install-gpu-operator-air-gapped.rst index 6ed322009..48c8c43d0 100644 --- a/gpu-operator/install-gpu-operator-air-gapped.rst +++ b/gpu-operator/install-gpu-operator-air-gapped.rst @@ -237,6 +237,29 @@ Sample of ``values.yaml`` for GPU Operator v1.9.0: imagePullSecrets: [] +DRA Driver Image +================ + +If you deploy the ``GPUCluster`` custom resource definition, mirror the DRA Driver for NVIDIA GPUs image in addition to the other GPU Operator images: + +.. code-block:: console + + $ docker pull registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v${dra_version} + +Tag and push this image to the local registry, then configure its location in ``values.yaml``: + +.. code-block:: yaml + + draDriver: + repository: / + image: dra-driver-nvidia-gpu + version: v${dra_version} + imagePullSecrets: [] + +Use the remaining ``GPUCluster`` installation settings from :doc:`DRA Driver for NVIDIA GPUs `. +The ``GPUCluster`` and ComputeDomain custom resource definitions are packaged in the GPU Operator chart and do not require a separate download during installation. + + ************************ Local Package Repository ************************ diff --git a/gpu-operator/uninstall.rst b/gpu-operator/uninstall.rst index 504315122..f47ebadb4 100644 --- a/gpu-operator/uninstall.rst +++ b/gpu-operator/uninstall.rst @@ -22,8 +22,28 @@ Uninstalling the GPU Operator Perform the following steps to uninstall the Operator. +#. If a ``GPUCluster`` resource exists, delete user workloads and user-created ResourceClaims for NVIDIA GPUs. + Wait for the workload pods to terminate and the claims to be unprepared. + + .. code-block:: console + + $ kubectl get gpucluster gpu-cluster + $ kubectl get resourceclaim --all-namespaces + + The Helm uninstall command later in this procedure runs a ``pre-delete`` hook that deletes ``gpu-cluster``. + The hook waits for the GPUCluster finalizer to remove Operator-managed ResourceClaim consumers before Helm removes the Operator. + #. Optional: List and delete NVIDIA driver custom resources. + If you use ``GPUCluster``, skip this optional step unless you first delete ``gpu-cluster`` and wait for its finalizer to complete: + + .. code-block:: console + + $ kubectl delete gpucluster gpu-cluster + $ kubectl wait --for=delete gpucluster/gpu-cluster --timeout=5m + + The chart-managed default ``NVIDIADriver`` resource does not need to be deleted separately. + .. code-block:: console $ kubectl get nvidiadrivers @@ -67,16 +87,27 @@ By default, Helm does not `support deleting existing CRDs `__ +To overcome this, the Operator uses a `pre-delete hook `__ to perform the CRD cleanup. The ``operator.cleanupCRD`` chart parameter is added to enable this hook. This parameter is disabled by default. You can enable the hook by specifying ``--set operator.cleanupCRD=true`` during install or upgrade to perform automatic CRD cleanup on chart deletion. +The cleanup hook removes the ``gpuclusters.nvidia.com`` CRD, but does not remove the two ComputeDomain CRDs. +After verifying that no ComputeDomain resources remain, you can remove those definitions manually: + +.. code-block:: console + + $ kubectl delete crd computedomains.resource.nvidia.com + $ kubectl delete crd computedomaincliques.resource.nvidia.com + Alternatively, you can delete the custom resource definition: .. code-block:: console @@ -95,3 +126,7 @@ Alternatively, you can delete the custom resource definition: * Helm hooks used with the GPU Operator use the Operator image itself. If the Operator image cannot be pulled successfully (either due to network error or an invalid NGC registry secret in case of NVAIE), hooks will fail. In this case, delete the chart and specify the ``--no-hooks`` argument to avoid hanging on hook failures. + + If a ``GPUCluster`` resource exists, do not specify ``--no-hooks``. + Doing so can remove the Operator before the GPUCluster finalizer completes ordered teardown of ResourceClaim-consuming operands. + First resolve the hook failure, or delete ``gpu-cluster`` and wait for it to disappear while the Operator is still running. diff --git a/gpu-operator/upgrade.rst b/gpu-operator/upgrade.rst index 383c74ecf..5e0d6aaf6 100644 --- a/gpu-operator/upgrade.rst +++ b/gpu-operator/upgrade.rst @@ -65,7 +65,7 @@ With this procedure, all existing GPU Operator resources are updated inline and $ export RELEASE_TAG=${version} -#. Apply the custom resource definitions for the cluster policy and NVIDIA driver: +#. Apply the custom resource definitions for the cluster policy, NVIDIA driver, and GPU cluster: .. code-block:: console @@ -75,12 +75,28 @@ With this procedure, all existing GPU Operator resources are updated inline and $ kubectl apply -f \ https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_gpuclusters.yaml + *Example Output* .. code-block:: output customresourcedefinition.apiextensions.k8s.io/clusterpolicies.nvidia.com configured customresourcedefinition.apiextensions.k8s.io/nvidiadrivers.nvidia.com created + customresourcedefinition.apiextensions.k8s.io/gpuclusters.nvidia.com created + +#. If you use :doc:`DRA Driver for NVIDIA GPUs `, apply the ComputeDomain custom resource definitions: + + .. code-block:: console + + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomains.yaml + + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomaincliques.yaml + + Apply both definitions before upgrading, including when ComputeDomain support is currently disabled. #. Apply the custom resource definition for Node Feature Discovery: @@ -143,6 +159,12 @@ Starting with GPU Operator v22.09, a ``pre-upgrade`` Helm `hook `_. \ No newline at end of file +For upgrading the GPU Operator when running in OpenShift, refer to the official OpenShift documentation on `upgrading installed operators `_.