From 4891e0b67d3b50b7d3d72b9975b48933e11ab667 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Mon, 10 Aug 2026 13:12:55 -0400 Subject: [PATCH 1/5] docs: DRA updates for 26.7.0 Signed-off-by: Mike McKiernan --- gpu-operator/dra-intro-install.rst | 497 +++++++++++------- gpu-operator/getting-started.rst | 23 + gpu-operator/gpu-driver-configuration.rst | 10 + gpu-operator/gpu-driver-upgrades.rst | 4 + .../install-gpu-operator-air-gapped.rst | 23 + gpu-operator/uninstall.rst | 37 +- gpu-operator/upgrade.rst | 26 +- 7 files changed, 422 insertions(+), 198 deletions(-) diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index 032ae0263..ca41b22a9 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -20,256 +20,271 @@ DRA Driver for NVIDIA GPUs ########################## -Dynamic Resource Allocation (DRA) is a Kubernetes concept for flexibly requesting, configuring, and sharing specialized devices like GPUs. -This page describes how to install and upgrade the DRA Driver for NVIDIA GPUs v${dra_version} with the NVIDIA GPU Operator. +Dynamic Resource Allocation (DRA) is a Kubernetes API for flexibly requesting, +configuring, and sharing specialized devices such as GPUs. +This page describes how to use the GPU Operator to install and manage DRA Driver for NVIDIA GPUs v${dra_version}. -Before using the DRA Driver for NVIDIA GPUs, it is recommended that you are familiar with the following: +Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the following documentation: -* `Upstream Kubernetes DRA documentation `_. -* `DRA Driver documentation `__. +* `Dynamic Resource Allocation `_ in the Kubernetes documentation. +* `DRA Driver for NVIDIA GPUs documentation `__. -***************** +.. important:: + + GPU Operator management of the DRA enablement stack is available as a technology preview. + Technology preview features are not supported in production environments and are not functionally complete. + +******** Overview -***************** +******** -With the DRA Driver for NVIDIA GPUs, your Kubernetes workloads can allocate and consume the following two types of resources: +The GPU Operator manages the DRA enablement stack through the ``nvidia.com/v1alpha1`` ``GPUCluster`` custom resource. +``GPUCluster`` is a cluster-scoped singleton and its name must be ``gpu-cluster``. +The Helm chart creates this resource when you set ``gpuCluster.deployCR=true``. -* GPU allocation: for controlled sharing and dynamic reconfiguration of GPUs. - This functionality replaces the traditional GPU allocation method used by the NVIDIA Kubernetes Device Plugin. -* ComputeDomains: an abstraction for secure `Multi-Node NVLink (MNNVL) `_ for NVIDIA GB200 and similar systems. +A cluster can have either a ``GPUCluster`` or a ``ClusterPolicy`` resource, but not both. +``ClusterPolicy`` manages the enablement stack based on the NVIDIA Kubernetes Device Plugin, +whereas ``GPUCluster`` manages the stack based on DRA. +Mixed Device Plugin and DRA stacks are not supported. -You can use these features independently or together in the same cluster. +The ``GPUCluster`` stack includes the following components: -.. _known-issues: +* The DRA driver GPU capability for ``gpu.nvidia.com``, ``mig.nvidia.com``, and ``vfio.gpu.nvidia.com`` devices. + This capability is always enabled. +* The ComputeDomain controller and kubelet plugin for Multi-Node NVLink (MNNVL). + ComputeDomain support is enabled by default and can be disabled. +* A DRA validator that allocates a GPU by using a ``ResourceClaim`` and verifies that the device is usable. +* DCGM Exporter, which is enabled by default. +* Standalone DCGM, which is disabled by default. -Known Issues -************ +The ``GPUCluster`` resource does not manage the NVIDIA GPU driver. +Use an ``NVIDIADriver`` resource to install a containerized driver or use a driver that is pre-installed on the host. +The Operator automatically assigns the ``nvidia.com/gpu.deploy.*`` labels that control operand placement. -This section covers known issues for the DRA Driver when used with the NVIDIA GPU Operator. -For known issues specific to the DRA Driver itself, refer to the `DRA Driver v${dra_version} release notes `__. +Limitations +=========== -* There is a known issue where the NVIDIA Driver Manager is not aware of the DRA driver kubelet plugin, and will not correctly evict it on pod restarts. - You must label the nodes you plan to use with DRA GPU allocation and pass the node label in the GPU Operator Helm command in the ``driver.manager.env`` flag. - This enables the NVIDIA Driver Manager to evict the GPU kubelet plugin correctly on driver container upgrades. -* For A100 GPUs, the MIG manager does not automatically evict the DRA kubelet plugin during MIG configuration changes. - If the DRA kubelet plugin is deployed before a MIG change, then you must manually restart the DRA kubelet plugin. +* Use this workflow for a new installation. + An in-place migration from a standalone DRA driver Helm release or from ``ClusterPolicy`` to ``GPUCluster`` is not supported. +* ``GPUCluster`` does not expose the controller affinity, priority class, toleration, + and kubelet-plugin node selector overrides that were used by the previous standalone DRA driver procedure for Google Kubernetes Engine. + This page does not provide a managed DRA installation procedure for GKE. ************* Prerequisites ************* +In addition to ensuring that your GPUs and cluster align with the :ref:`GPU Operator support matrix `, verify the following prerequisites: -In addition to ensuring your GPUs and cluster align with the :ref:`GPU Operator support matrix `, the following prerequisites must be met: +* Kubernetes v1.34.2 or later with a ``resource.k8s.io`` ``DeviceClass`` API available. +* NVIDIA GPU driver version 580 or later. +* An underlying container runtime that supports CDI and is configured to use CDI for injecting DRA-allocated devices. +* No ``ClusterPolicy`` resource exists in the cluster. -* Kubernetes v1.34.2 or later. + .. code-block:: console - .. note:: - If you plan to use traditional extended resource requests such as ``nvidia.com/gpu`` alongside the DRA driver, you must enable the `DRAExtendedResource `_ feature gate. - This enabled by default in Kubernetes v1.36.0 and later. - It allows the scheduler to translate extended resource requests into ResourceClaims for the DRA driver. - -* For GPU allocation support, you must label nodes that will support GPU allocation with ``nvidia.com/dra-kubelet-plugin=true`` and use this label as a node selector in the DRA driver Helm chart. - This is required to avoid the :ref:`known issue ` when using the GPU Operator with the DRA Driver. - Steps for labeling nodes are provided in the install section. - The label is also passed to the GPU Operator Helm command via the ``driver.manager.env`` flag. - -.. _computedomain-prereqs: + $ kubectl get clusterpolicy -* For ComputeDomain ensure the following: + If a ``ClusterPolicy`` exists, use a new cluster for the ``GPUCluster`` workflow. - * NVIDIA Grace Blackwell GPUs with Multi-Node NVLink (MNNVL) available on your cluster. - For example, NVIDIA HGX GB200 NVL72 or NVIDIA HGX GB300 NVL72. - Refer to the `NVIDIA Multi-Node NVLink Systems documentation `_ for details on Multi-Node NVLink systems. - - * For using ComputeDomains with a pre-installed GPU Driver: - - * The corresponding nvidia-imex-* packages installed through your Linux distribution's package manager. - * The IMEX systemd service disabled before installing the GPU Operator (on all GPU nodes). - For example: +.. note:: - .. code-block:: console + To use traditional extended resource requests such as ``nvidia.com/gpu`` with the DRA driver, + enable the `DRAExtendedResource `_ feature gate. + This feature gate is enabled by default in Kubernetes v1.36.0 and later. - $ systemctl disable --now nvidia-imex.service && systemctl mask nvidia-imex.service +.. _computedomain-prereqs: +For ComputeDomain support, verify the following additional prerequisites: -.. note:: +* NVIDIA Grace Blackwell GPUs with Multi-Node NVLink available on your cluster, + such as NVIDIA HGX GB200 NVL72 or NVIDIA HGX GB300 NVL72. + Refer to the `NVIDIA Multi-Node NVLink Systems documentation `_ for more information. +* If you use a pre-installed GPU driver, install the corresponding ``nvidia-imex-*`` packages through the Linux distribution package manager. +* If you use a pre-installed GPU driver, disable and mask the IMEX systemd service on every GPU node before installing the GPU Operator: - Installing GPU Operator ${version} configures some additional DRA Driver for NVIDIA GPUs prerequisites for you: - - * Container Device Interface (CDI) enabled in the underlying container runtime (such as containerd or CRI-O) - * NVIDIA Driver version 580 or later. - * Deploy Node Feature Discovery (NFD) and GPU Feature Discovery (GFD) on your cluster. + .. code-block:: console + $ systemctl disable --now nvidia-imex.service + $ systemctl mask nvidia-imex.service ******* Install ******* -This section covers fresh installs of the GPU Operator and DRA Driver for NVIDIA GPUs. -If you are upgrading an earlier version of the DRA Driver for NVIDIA GPUs, refer to the :ref:`Upgrade ` section. - -.. note:: - The ``nvidiaDriverRoot`` flag sets the root directory for the NVIDIA GPU driver. - The default value is ``/``, which is typical for drivers installed directly on the host. - With GPU Operator–managed drivers (default), drivers are installed to ``/run/nvidia/driver``. - If you are using `pre-installed drivers `_, remove the ``nvidiaDriverRoot`` flag or set it to ``/``. +The following procedures install the GPU Operator and create the ``gpu-cluster`` resource. +The Operator deploys the DRA driver operands in the GPU Operator namespace. +#. Add the NVIDIA Helm repository: -.. tab-set:: - :sync-group: dra - - .. tab-item:: GPU Allocation - :sync: gpu-allocation + .. code-block:: console - 1. Label every node that will support GPU allocation through DRA: + $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ + && helm repo update - .. code-block:: console +#. Install the GPU Operator by using the procedure for your driver configuration. - $ kubectl label node $HOSTNAME nvidia.com/dra-kubelet-plugin=true + .. tab-set:: + :sync-group: dra-driver-install - 2. Add the NVIDIA Helm repository: + .. tab-item:: Operator-Managed Driver + :sync: managed .. code-block:: console - $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ - && helm repo update + $ helm upgrade --install gpu-operator nvidia/gpu-operator \ + --version=${version} \ + --namespace gpu-operator \ + --create-namespace \ + --set clusterPolicy.deployCR=false \ + --set gpuCluster.deployCR=true \ + --set driver.nvidiaDriverCRD.enabled=true + + The chart creates the default ``NVIDIADriver`` resource to manage the NVIDIA GPU driver. - 3. Install the GPU Operator with the NVIDIA Kubernetes Device Plugin disabled: + .. tab-item:: Pre-Installed Driver + :sync: preinstalled .. code-block:: console $ helm upgrade --install gpu-operator nvidia/gpu-operator \ --version=${version} \ - --create-namespace \ --namespace gpu-operator \ - --set devicePlugin.enabled=false \ - --set driver.manager.env[0].name=NODE_LABEL_FOR_GPU_POD_EVICTION \ - --set driver.manager.env[0].value="nvidia.com/dra-kubelet-plugin" + --create-namespace \ + --set clusterPolicy.deployCR=false \ + --set gpuCluster.deployCR=true \ + --set driver.enabled=false - Make sure the value of ``driver.manager.env`` matches the node label applied in step 1. +By default, the Operator enables both GPU allocation and ComputeDomain support. +To install GPU allocation without ComputeDomain support, add the following option to the Helm command: - Make sure the ``devicePlugin.enabled`` flag is set to ``false`` to disable the NVIDIA Kubernetes Device Plugin. - The DRA Driver for NVIDIA GPUs will be used to allocate GPUs. +.. code-block:: console - Refer to the `GPU Operator installation guide `_ for additional configuration options. - If you plan to use MIG devices, refer to the `GPU Operator MIG documentation `_ to configure your cluster for MIG support. - Restart the DRA driver pod to discover MIG devices created by the GPU Operator MIG Manager. + --set draDriver.computeDomains.enabled=false - 4. Create a ``values.yaml`` file for the DRA driver: +Do not install a separate ``dra-driver-nvidia-gpu`` Helm release for this managed workflow. - .. code-block:: yaml +*********************** +Configure the DRA Stack +*********************** - image: - pullPolicy: IfNotPresent - kubeletPlugin: - nodeSelector: - nvidia.com/dra-kubelet-plugin: "true" +The GPU Operator Helm values render the specification of the ``gpu-cluster`` resource. +The following settings provide the primary configuration surface: - If you are using Google Kubernetes Engine (GKE), the DRA driver requires additional overrides for the driver root, controller affinity, and tolerations: +.. list-table:: + :header-rows: 1 + :widths: 38 62 - .. code-block:: yaml + * - Helm value + - Description - # GKE helm values example - # "/home/kubernetes/bin/nvidia" is the default driver root on GKE. - nvidiaDriverRoot: "/home/kubernetes/bin/nvidia" + * - ``draDriver.repository``, ``draDriver.image``, and ``draDriver.version`` + - Configure the DRA driver container image. - controller: - priorityClassName: "" - affinity: null - image: - pullPolicy: IfNotPresent - kubeletPlugin: - priorityClassName: "" - tolerations: - - effect: NoSchedule - key: nvidia.com/gpu - operator: Exists - nodeSelector: - nvidia.com/dra-kubelet-plugin: "true" + * - ``draDriver.imagePullPolicy`` and ``draDriver.imagePullSecrets`` + - Configure image pulling for the DRA driver containers. - 5. Install the DRA driver: + * - ``draDriver.featureGates`` + - Enable or disable DRA driver feature gates. + The Operator renders the map as the ``FEATURE_GATES`` environment variable for DRA driver containers. - .. code-block:: console + * - ``draDriver.gpus.kubeletPlugin`` + - Configure environment variables, resource requests and limits, and the gRPC health check for the GPU kubelet-plugin container. - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set nvidiaDriverRoot=/run/nvidia/driver \ - --set gpuResourcesEnabledOverride=true \ - -f values.yaml + * - ``draDriver.computeDomains.enabled`` + - Enable or disable the ComputeDomain controller and kubelet-plugin container. - For GKE, omit ``--set nvidiaDriverRoot=/run/nvidia/driver``; the value comes from the GKE ``values.yaml`` file. + * - ``draDriver.computeDomains.controller`` + - Configure environment variables and resource requests and limits for the ComputeDomain controller. - .. tab-item:: ComputeDomain - :sync: computedomain + * - ``draDriver.computeDomains.kubeletPlugin`` + - Configure environment variables, resource requests and limits, and the gRPC health check for the ComputeDomain kubelet-plugin container. - 1. Add the NVIDIA Helm repository: + * - ``hostPaths.kubeletRootDir`` + - Configure the kubelet root directory when it differs from ``/var/lib/kubelet``. - .. code-block:: console + * - ``daemonsets`` + - Configure common labels, annotations, and tolerations. + ``daemonsets.priorityClassName`` applies to the DRA driver kubelet plugin and ComputeDomain controller. - $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update +For example, the following values enable a DRA driver feature gate, configure container resources, +and enable the default gRPC health checks on their default ports: - 2. Install the GPU Operator: +.. code-block:: yaml - .. code-block:: console + draDriver: + featureGates: + NVMLDeviceHealthCheck: true + gpus: + kubeletPlugin: + resources: + requests: + cpu: 50m + memory: 64Mi + healthcheck: + enabled: true + computeDomains: + enabled: true + controller: + resources: + requests: + cpu: 50m + memory: 64Mi + kubeletPlugin: + healthcheck: + enabled: true - $ helm upgrade --install gpu-operator nvidia/gpu-operator \ - --version=${version} \ - --create-namespace \ - --namespace gpu-operator +The GPU kubelet-plugin health-check port defaults to ``51516`` and the ComputeDomain kubelet-plugin health-check port defaults to ``51515``. +The Operator also enables ComputeDomain GPU clique labeling by setting ``GPU_CLIQUE_LABEL_ENABLED=true`` automatically. - Refer to the `GPU Operator installation guide `_ for additional configuration options. +********************* +Validate Installation +********************* - 3. Install the DRA driver. +Reconciliation typically completes within 3 minutes. +During reconciliation, the STATUS column progresses from empty to `notReady` to `ready`. - Example for a GPU Operator–managed driver: +#. Verify that the ``GPUCluster`` resource is ready: - .. code-block:: console + .. code-block:: console - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set resources.gpus.enabled=false \ - --set nvidiaDriverRoot=/run/nvidia/driver + $ kubectl get gpucluster gpu-cluster - Example for a pre-installed GPU driver: + *Example Output* - .. code-block:: console + .. code-block:: output - $ helm upgrade -i dra-driver-nvidia-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --create-namespace \ - --set resources.gpus.enabled=false + NAME STATUS AGE + gpu-cluster ready 3m -********************* -Validate Installation -********************* - -1. Confirm that the DRA driver components are running: + If the resource does not become ready, inspect its conditions and recent events: .. code-block:: console - $ kubectl get pods -n nvidia-dra-driver-gpu + $ kubectl describe gpucluster gpu-cluster - *Example Output* + A ``PrerequisiteNotMet`` condition can indicate that a ``ClusterPolicy`` resource exists. + An ``OperandNotReady`` condition indicates that the Operator is waiting for one or more managed pods. - .. code-block:: output +#. Confirm that the managed components are running in the GPU Operator namespace: - NAME READY STATUS RESTARTS AGE - dra-driver-nvidia-gpu-controller-67cb99d84b-5q7kj 1/1 Running 0 7m26s - dra-driver-nvidia-gpu-kubelet-plugin-h5xsn 2/2 Running 0 7m27s + .. code-block:: console - The controller pod runs the ComputeDomain controller (1 container). The kubelet-plugin pod runs two containers, one for GPU resources (gpus) and one for ComputeDomain resources (compute-domains), so it shows ``2/2`` when both are enabled. One kubelet-plugin pod appears per GPU node. + $ kubectl get pods -n gpu-operator - If you installed with ``--set resources.computeDomains.enabled=false``, the controller pod is not present and the kubelet-plugin pod shows ``1/1``. The same is true if you disabled GPU allocation during install. + Expected workload names include the following: - .. note:: - If you upgraded an existing v25.x installation, the pod names retain the ``nvidia-dra-driver-gpu-`` prefix (for example, ``nvidia-dra-driver-gpu-controller-*``) because the upgrade preserves the original resource names through the ``nameOverride`` flag. + * ``nvidia-dra-driver-kubelet-plugin`` + * ``nvidia-dra-driver-controller`` when ComputeDomain support is enabled + * ``nvidia-dra-validator`` + * ``nvidia-dcgm-exporter-dra`` when DCGM Exporter is enabled + * ``nvidia-dcgm-dra`` when standalone DCGM is enabled -2. Verify that GPU DeviceClasses are available: + The DRA kubelet-plugin DaemonSet runs a GPU container and, + when ComputeDomain support is enabled, a ComputeDomain container. + The DRA validator and enabled telemetry operands run one pod on each GPU node. + +#. Verify that the DeviceClasses are available: .. code-block:: console @@ -279,62 +294,154 @@ Validate Installation .. code-block:: output - NAME AGE - compute-domain-daemon.nvidia.com 55s - compute-domain-default-channel.nvidia.com 55s - gpu.nvidia.com 55s - mig.nvidia.com 55s + compute-domain-daemon.nvidia.com 2m + compute-domain-default-channel.nvidia.com 2m + gpu.nvidia.com 2m + mig.nvidia.com 2m + vfio.gpu.nvidia.com 2m + + The ComputeDomain DeviceClasses are present only when ComputeDomain support is enabled. + +Additional validation procedures are available in the upstream DRA Driver documentation: + +* `Run a sample ComputeDomain workload `__. +* `Run a sample GPU allocation workload `__. -The ``compute-domain-daemon.nvidia.com`` and ``compute-domain-default-channel.nvidia.com`` DeviceClasses are installed when ComputeDomain support is enabled. -The ``gpu.nvidia.com`` and ``mig.nvidia.com`` DeviceClasses are installed when GPU allocation support is enabled. +********* +Telemetry +********* -Additional validation steps are available in the upstream DRA Driver documentation: +DCGM Exporter is enabled by default for the ``GPUCluster`` stack and uses its embedded host engine. +Set ``dcgm.enabled=true`` to deploy the standalone ``nvidia-dcgm-dra`` host engine instead. -* `Validate setup for ComputeDomain allocation `_ -* `Validate setup for GPU allocation `_ +When either ``dcgmExporter.enablePodLabels`` or ``dcgmExporter.enablePodUID`` is enabled, +the Operator enables DRA ``ResourceSlice`` attribution in DCGM Exporter and grants the exporter read access to ``ResourceSlice`` objects. +This enables pod metadata enrichment for GPUs allocated through DRA. +.. _dra-driver-upgrade: .. _upgrade: ******* Upgrade ******* -Starting with v0.4.0, the DRA Driver for NVIDIA GPUs moved to ``kubernetes-sigs/dra-driver-nvidia-gpu`` and adopted semantic versioning. -The Helm chart was renamed from ``nvidia-dra-driver-gpu`` to ``dra-driver-nvidia-gpu`` and is published to new NGC Helm and container registries. +The DRA driver is upgraded as part of the GPU Operator release. +When you upgrade the GPU Operator, preserve ``clusterPolicy.deployCR=false`` and ``gpuCluster.deployCR=true`` in your values. +You can select a different DRA driver version by setting ``draDriver.version``. +Refer to :ref:`Upgrading the NVIDIA GPU Operator` for the Operator upgrade procedure and required CRD updates. -When upgrading from v25.x you must explicitly set ``nameOverride`` and ``--version`` to avoid creating duplicate Kubernetes manifests under different names. -Without ``--set nameOverride=nvidia-dra-driver-gpu``, the upgrade creates new daemonsets and deployments under the new chart name instead of upgrading the existing resources in place. +During an Operator-managed NVIDIA GPU driver upgrade, pods with allocated ``gpu.nvidia.com`` ResourceClaims are treated as GPU workloads. +The driver upgrade policy applies to those pods, and the DRA kubelet plugin remains available while claims are unprepared. +You do not need to configure a separate node label for DRA workload eviction. -.. important:: - After upgrading to v0.4.0, downgrading to v25.x is not supported. +This procedure does not migrate a standalone DRA driver Helm release to ``GPUCluster``. +For a standalone installation, refer to the `upstream DRA driver upgrade guide `__ +or select the GPU Operator documentation version that matches the installed release. + +********* +Uninstall +********* + +Before uninstalling the GPU Operator, delete user workloads and user-created ResourceClaims for GPUs +and verify that the workload pods terminate and the claims are unprepared. +Afterward, uninstall the GPU Operator by using Helm. -Upgrade from v25.x to v0.4.0 or later -************************************** +The chart includes a ``pre-delete`` hook that deletes ``gpu-cluster`` and waits for the GPUCluster finalizer to perform ordered teardown. -1. Apply the v${dra_version} CRDs for ComputeDomains and ComputeDomainsCliques before upgrading the Helm chart. - Refer to the `v${dra_version} release page `__ for the CRD manifests. +.. tip:: The uninstall command output can report that the ``gpu-cluster`` resource was kept. -2. Run ``helm upgrade`` with ``nameOverride`` and ``--version``, preserving any original install flags such as ``gpuResourcesEnabledOverride`` and ``nvidiaDriverRoot``: + However, a pre-delete hook actually deletes the resource. .. code-block:: console - $ helm upgrade -i nvidia-dra-driver-gpu nvidia/dra-driver-nvidia-gpu \ - --version=${dra_version} \ - --namespace nvidia-dra-driver-gpu \ - --set nameOverride=nvidia-dra-driver-gpu \ - --set gpuResourcesEnabledOverride=true \ - --set nvidiaDriverRoot=/run/nvidia/driver + These resources were kept due to the resource policy: + [GPUCluster] gpu-cluster + + release "gpu-operator" uninstalled + +The finalizer removes Operator-managed DaemonSets that consume ResourceClaims before the DRA kubelet plugin is removed. +Do not use ``helm uninstall --no-hooks`` while ``gpu-cluster`` exists because Helm can remove the Operator before this teardown completes. + +Refer to :doc:`uninstall` for the complete uninstall procedure and CRD cleanup information. + +*************** +Troubleshooting +*************** + +Recover From an Uninstall That Skipped the Pre-Delete Hook +========================================================== + +If you run ``helm uninstall gpu-operator --no-hooks`` while ``gpu-cluster`` exists, +Helm removes the Operator before the ``pre-delete`` hook can perform an ordered teardown. +The following resources are left in the cluster with no controller to reconcile them: + +* The ``gpu-cluster`` resource, which cannot be deleted because the + ``gpucluster.nvidia.com/dra-resourceclaim`` finalizer requires the Operator. +* The DRA driver kubelet plugin, DRA validator, and DCGM Exporter pods. +* Operator-managed ResourceClaims for the preceding pods. -3. Verify the upgrade: +To recover, remove the finalizer from ``gpu-cluster``, then force-delete the orphaned operands: + +#. Remove the finalizer from ``gpu-cluster``: + + .. code-block:: console + + $ kubectl patch gpucluster gpu-cluster --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' + +#. Remove finalizers from the orphaned ResourceClaims in the GPU Operator namespace: + + .. code-block:: console + + $ for rc in $(kubectl get resourceclaim -n gpu-operator -o name); do + kubectl patch $rc -n gpu-operator --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' + done + +#. Force-delete any pods that remain in the ``Terminating`` state: + + .. code-block:: console + + $ kubectl delete pods -n gpu-operator --all --grace-period=0 --force + +#. Delete the GPU Operator namespace to remove residual state: .. code-block:: console - $ kubectl get pods -n nvidia-dra-driver-gpu + $ kubectl delete namespace gpu-operator + +To avoid this recovery path, run ``helm uninstall`` without ``--no-hooks`` so that the +``pre-delete`` hook can complete ordered teardown. + +Clean Up ResourceClaims Stuck in ``deleted,allocated,reserved`` +=============================================================== + +ResourceClaims in the ``deleted,allocated,reserved`` state indicate that the API server +accepted a delete request, but the DRA driver kubelet plugin has not completed the +device _unprepare_ step for the claim. +This state can persist if the DRA driver kubelet plugin was removed or restarted +before it could unprepare the allocated devices. + +Take one of the following actions: + +* If the DRA driver kubelet plugin can be restarted, ensure that the + ``nvidia-dra-driver-kubelet-plugin`` pod is running. + The plugin completes the unprepare step and the ResourceClaims are removed. + +* If the DRA driver kubelet plugin cannot be restarted, such as after an + uninstall that skipped the pre-delete hook, remove the finalizers from + the affected ResourceClaims: + + .. code-block:: console - All controller and kubelet-plugin pods should reach ``Running`` status, and existing ResourceClaims should remain in the ``allocated,reserved`` state. + $ for rc in $(kubectl get resourceclaim -n gpu-operator -o name); do + kubectl patch $rc -n gpu-operator --type=json \ + -p='[{"op":"remove","path":"/metadata/finalizers"}]' + done -Refer to the `upstream upgrade guide `__ for additional detail. + Removing finalizers bypasses the unprepare step. + Use this option only during recovery when no DRA workloads are running. ************************ Additional Documentation @@ -342,5 +449,5 @@ Additional Documentation For more details on the DRA Driver for NVIDIA GPUs, refer to the following resources: -* `DRA Driver for NVIDIA GPUs documentation `__ -* `DRA Driver v${dra_version} release notes `__ \ No newline at end of file +* `DRA Driver for NVIDIA GPUs documentation `__. +* `DRA Driver v${dra_version} release notes `__. diff --git a/gpu-operator/getting-started.rst b/gpu-operator/getting-started.rst index 1f26676a1..98df7f922 100644 --- a/gpu-operator/getting-started.rst +++ b/gpu-operator/getting-started.rst @@ -362,6 +362,11 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, the Operator deploys NVIDIA Confidential Computing Manager for Kubernetes. - ``false`` + * - ``clusterPolicy.deployCR`` + - When set to ``true``, the chart creates the ``ClusterPolicy`` resource for the NVIDIA Kubernetes Device Plugin enablement stack. + Set this value to ``false`` when using ``gpuCluster.deployCR=true``. + - ``true`` + * - ``cdi.enabled`` - When set to ``true`` (default), the Container Device Interface (CDI) will be used for injecting GPUs into workload containers. @@ -416,6 +421,7 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, Kubernetes pod labels are added as Prometheus label dimensions on the GPU metrics. Enabling this option causes the Operator to provision a cluster-scoped ClusterRole and ClusterRoleBinding (``nvidia-dcgm-exporter-read-pods``) that grants the DCGM Exporter service account ``get``, ``list``, and ``watch`` access to pods. + For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects. Use ``dcgmExporter.podLabelAllowlistRegex`` to limit which labels are emitted. - ``false`` @@ -423,6 +429,7 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - When set to ``true``, the Kubernetes pod UID is added as a Prometheus label dimension on the GPU metrics. Like ``dcgmExporter.enablePodLabels``, this provisions a cluster-scoped ClusterRole and ClusterRoleBinding that grants the DCGM Exporter service account ``get``, ``list``, and ``watch`` access to pods. + For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects. - ``false`` * - ``dcgmExporter.podLabelAllowlistRegex`` @@ -471,6 +478,21 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. custom driver images. - ``nvcr.io/nvidia`` + * - ``draDriver.computeDomains.enabled`` + - When set to ``true``, the ``GPUCluster`` stack deploys the ComputeDomain controller and kubelet plugin. + GPU allocation through DRA is always enabled when a ``GPUCluster`` is deployed. + Refer to :doc:`DRA Driver for NVIDIA GPUs ` for more information. + - ``true`` + + * - ``draDriver.version`` + - Version of the DRA Driver for NVIDIA GPUs deployed by the ``GPUCluster`` stack. + - ``v${dra_version}`` + + * - ``gpuCluster.deployCR`` + - When set to ``true``, the chart creates the technology preview ``GPUCluster`` resource for the DRA enablement stack. + Set ``clusterPolicy.deployCR=false`` and refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete installation procedure. + - ``false`` + * - ``driver.rdma.enabled`` - Controls whether the driver daemon set builds and loads the legacy ``nvidia-peermem`` kernel module. @@ -877,6 +899,7 @@ After verifying the installation, you can configure the GPU Operator for your wo - :doc:`gpu-sharing` — Share a single GPU across multiple pods using time-slicing or MPS. - :doc:`gpu-operator-mig` — Configure Multi-Instance GPU (MIG) partitioning on supported GPUs. - :doc:`gpu-operator-rdma` — Enable GPUDirect RDMA for high-performance networking. +- :doc:`dra-intro-install` — Allocate GPUs by using Kubernetes Dynamic Resource Allocation (DRA). - :doc:`gpu-driver-configuration` — Use the NVIDIA GPU Driver Custom Resource Definition to manage drivers per node. - :doc:`precompiled-drivers` — Speed up driver deployments with precompiled kernel modules. - :doc:`cdi` — Learn about Container Device Interface (CDI) and NRI Plugin mode. diff --git a/gpu-operator/gpu-driver-configuration.rst b/gpu-operator/gpu-driver-configuration.rst index 7fd8b68f4..46f3f85df 100644 --- a/gpu-operator/gpu-driver-configuration.rst +++ b/gpu-operator/gpu-driver-configuration.rst @@ -30,6 +30,10 @@ You can create one or more instances of an NVIDIA driver (``NVIDIADriver``) cust to specify the NVIDIA GPU driver type and driver version to configure on specific nodes. You can specify labels in the node selector field to control which NVIDIA driver configuration is applied to specific nodes. +The technology preview ``GPUCluster`` enablement stack uses ``NVIDIADriver`` resources for an Operator-managed driver. +``GPUCluster`` manages the DRA operands and waits for the matching NVIDIA driver deployment to become ready. +Refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete ``GPUCluster`` installation procedure. + Driver Management Modes ======================= @@ -406,6 +410,12 @@ Installing the NVIDIA GPU Operator Perform the following steps to install the GPU Operator and use the NVIDIA driver custom resources. +.. note:: + + To use an ``NVIDIADriver`` resource with the ``GPUCluster`` DRA stack, + follow the installation procedure in :doc:`DRA Driver for NVIDIA GPUs `. + That procedure configures the chart to create ``GPUCluster`` instead of ``ClusterPolicy``. + #. Optional: If you want to run more than one driver type or version in the cluster, label the worker nodes to identify the driver type and version to install on each node: diff --git a/gpu-operator/gpu-driver-upgrades.rst b/gpu-operator/gpu-driver-upgrades.rst index 7185b7c60..1a1e98e45 100644 --- a/gpu-operator/gpu-driver-upgrades.rst +++ b/gpu-operator/gpu-driver-upgrades.rst @@ -46,6 +46,10 @@ The GPU Operator supports several methods for managing and automating this drive The GPU Operator only manages the lifecycle of containerized drivers. Drivers which are pre-installed on the host are not managed by the GPU Operator. +For the ``GPUCluster`` DRA stack, the Operator treats pods with an allocated +``gpu.nvidia.com`` ResourceClaim as GPU workloads during a driver upgrade. +The configured pod deletion policy applies to these workloads before the driver is reloaded. +The DRA kubelet plugin remains available while terminating pods unprepare their claims. ************************************ Upgrades with the Upgrade Controller diff --git a/gpu-operator/install-gpu-operator-air-gapped.rst b/gpu-operator/install-gpu-operator-air-gapped.rst index 6ed322009..f4180faff 100644 --- a/gpu-operator/install-gpu-operator-air-gapped.rst +++ b/gpu-operator/install-gpu-operator-air-gapped.rst @@ -237,6 +237,29 @@ Sample of ``values.yaml`` for GPU Operator v1.9.0: imagePullSecrets: [] +DRA Driver Image +================ + +If you deploy the technology preview ``GPUCluster`` custom resource definition, mirror the DRA Driver for NVIDIA GPUs image in addition to the other GPU Operator images: + +.. code-block:: console + + $ docker pull registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v${dra_version} + +Tag and push this image to the local registry, then configure its location in ``values.yaml``: + +.. code-block:: yaml + + draDriver: + repository: / + image: dra-driver-nvidia-gpu + version: v${dra_version} + imagePullSecrets: [] + +Use the remaining ``GPUCluster`` installation settings from :doc:`DRA Driver for NVIDIA GPUs `. +The ``GPUCluster`` and ComputeDomain custom resource definitions are packaged in the GPU Operator chart and do not require a separate download during installation. + + ************************ Local Package Repository ************************ diff --git a/gpu-operator/uninstall.rst b/gpu-operator/uninstall.rst index 504315122..f47ebadb4 100644 --- a/gpu-operator/uninstall.rst +++ b/gpu-operator/uninstall.rst @@ -22,8 +22,28 @@ Uninstalling the GPU Operator Perform the following steps to uninstall the Operator. +#. If a ``GPUCluster`` resource exists, delete user workloads and user-created ResourceClaims for NVIDIA GPUs. + Wait for the workload pods to terminate and the claims to be unprepared. + + .. code-block:: console + + $ kubectl get gpucluster gpu-cluster + $ kubectl get resourceclaim --all-namespaces + + The Helm uninstall command later in this procedure runs a ``pre-delete`` hook that deletes ``gpu-cluster``. + The hook waits for the GPUCluster finalizer to remove Operator-managed ResourceClaim consumers before Helm removes the Operator. + #. Optional: List and delete NVIDIA driver custom resources. + If you use ``GPUCluster``, skip this optional step unless you first delete ``gpu-cluster`` and wait for its finalizer to complete: + + .. code-block:: console + + $ kubectl delete gpucluster gpu-cluster + $ kubectl wait --for=delete gpucluster/gpu-cluster --timeout=5m + + The chart-managed default ``NVIDIADriver`` resource does not need to be deleted separately. + .. code-block:: console $ kubectl get nvidiadrivers @@ -67,16 +87,27 @@ By default, Helm does not `support deleting existing CRDs `__ +To overcome this, the Operator uses a `pre-delete hook `__ to perform the CRD cleanup. The ``operator.cleanupCRD`` chart parameter is added to enable this hook. This parameter is disabled by default. You can enable the hook by specifying ``--set operator.cleanupCRD=true`` during install or upgrade to perform automatic CRD cleanup on chart deletion. +The cleanup hook removes the ``gpuclusters.nvidia.com`` CRD, but does not remove the two ComputeDomain CRDs. +After verifying that no ComputeDomain resources remain, you can remove those definitions manually: + +.. code-block:: console + + $ kubectl delete crd computedomains.resource.nvidia.com + $ kubectl delete crd computedomaincliques.resource.nvidia.com + Alternatively, you can delete the custom resource definition: .. code-block:: console @@ -95,3 +126,7 @@ Alternatively, you can delete the custom resource definition: * Helm hooks used with the GPU Operator use the Operator image itself. If the Operator image cannot be pulled successfully (either due to network error or an invalid NGC registry secret in case of NVAIE), hooks will fail. In this case, delete the chart and specify the ``--no-hooks`` argument to avoid hanging on hook failures. + + If a ``GPUCluster`` resource exists, do not specify ``--no-hooks``. + Doing so can remove the Operator before the GPUCluster finalizer completes ordered teardown of ResourceClaim-consuming operands. + First resolve the hook failure, or delete ``gpu-cluster`` and wait for it to disappear while the Operator is still running. diff --git a/gpu-operator/upgrade.rst b/gpu-operator/upgrade.rst index 383c74ecf..5e0d6aaf6 100644 --- a/gpu-operator/upgrade.rst +++ b/gpu-operator/upgrade.rst @@ -65,7 +65,7 @@ With this procedure, all existing GPU Operator resources are updated inline and $ export RELEASE_TAG=${version} -#. Apply the custom resource definitions for the cluster policy and NVIDIA driver: +#. Apply the custom resource definitions for the cluster policy, NVIDIA driver, and GPU cluster: .. code-block:: console @@ -75,12 +75,28 @@ With this procedure, all existing GPU Operator resources are updated inline and $ kubectl apply -f \ https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_gpuclusters.yaml + *Example Output* .. code-block:: output customresourcedefinition.apiextensions.k8s.io/clusterpolicies.nvidia.com configured customresourcedefinition.apiextensions.k8s.io/nvidiadrivers.nvidia.com created + customresourcedefinition.apiextensions.k8s.io/gpuclusters.nvidia.com created + +#. If you use :doc:`DRA Driver for NVIDIA GPUs `, apply the ComputeDomain custom resource definitions: + + .. code-block:: console + + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomains.yaml + + $ kubectl apply -f \ + https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomaincliques.yaml + + Apply both definitions before upgrading, including when ComputeDomain support is currently disabled. #. Apply the custom resource definition for Node Feature Discovery: @@ -143,6 +159,12 @@ Starting with GPU Operator v22.09, a ``pre-upgrade`` Helm `hook `_. \ No newline at end of file +For upgrading the GPU Operator when running in OpenShift, refer to the official OpenShift documentation on `upgrading installed operators `_. From 0c9c802d0dbff7ac67c04e3b47e1ce90d0117ca9 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Fri, 14 Aug 2026 11:44:24 -0400 Subject: [PATCH 2/5] docs: comp with device plugin Signed-off-by: Mike McKiernan --- gpu-operator/dra-intro-install.rst | 160 ++++++++++++++++++++++++++--- 1 file changed, 146 insertions(+), 14 deletions(-) diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index ca41b22a9..c80dc4976 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -31,23 +31,153 @@ Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the follo .. important:: - GPU Operator management of the DRA enablement stack is available as a technology preview. + GPU Operator management of DRA is available as a technology preview. Technology preview features are not supported in production environments and are not functionally complete. +********************************* +Comparison: DRA and Device Plugin +********************************* + +The DRA Driver for NVIDIA GPUs and the NVIDIA Kubernetes Device Plugin provide alternative mechanisms for allocating NVIDIA GPU resources. +The mechanisms do not provide feature parity. +A cluster can have either a ``GPUCluster`` resource for DRA or a ``ClusterPolicy`` resource for the Device Plugin, but not both. + +Use DRA for workloads that have the following requirements: + +* Coordinate Multi-Node NVLink workloads by using ComputeDomains. +* Use attribute-based GPU selection or allocation-specific device configuration instead of node-wide configuration. +* Configure dynamic NVIDIA Multi-Instance GPU (MIG), CUDA Multi-Process Service (MPS), + CUDA time-slicing, or Virtual Function I/O (VFIO) passthrough for individual allocations. + +Use the Device Plugin for workloads that have the following requirements: + +* Request ``nvidia.com/gpu`` or MIG extended resources from existing Pod specifications and tools. +* Schedule CUDA time-slicing or MPS replicas as independent extended resources. +* Use the broader set of components managed through ``ClusterPolicy``, Device Plugin health reporting, + or Kubernetes Pod priority and preemption. + +Capability Comparison +===================== + +The following table compares the GPU allocation capabilities of the two mechanisms when managed by the GPU Operator: + +.. list-table:: GPU Allocation Capability Comparison + :header-rows: 1 + :widths: 23 38 39 + + * - Capability + - DRA with ``GPUCluster`` + - Device Plugin with ``ClusterPolicy`` + + * - API and Device Selection + - Uses DRA API objects for structured device selection and per-claim configuration. + - Uses Kubernetes extended resource names and counts. + Configuration applies at the node or cluster level. + + * - Full GPU and MIG Device Allocation + - Allocates full GPU devices and existing MIG devices. + Alpha ``DynamicMIG`` creates MIG devices for a claim. + - Allocates full GPU devices or preconfigured MIG devices. + MIG Manager configures MIG geometry before allocation. + + * - GPU Sharing + - Alpha ``TimeSlicingSettings`` and ``MPSSupport`` configure sharing through DRA claims. + - Advertises time-slicing replicas and experimental MPS replicas as independent extended resources. + + * - Multi-Node NVLink + - Provides the ComputeDomain API and controller for coordinated workloads. + - Does not provide the ComputeDomain API. + + * - Device Health Reporting + - Alpha ``NVMLDeviceHealthCheck`` is disabled by default. + gRPC health probes report kubelet plugin availability only. + - Reports unhealthy devices through the Kubernetes Device Plugin API. + + * - Device Injection and Managed Components + - Requires a Container Device Interface (CDI)-compatible runtime. + ``GPUCluster`` manages DRA, ComputeDomains, NVIDIA Data Center GPU Manager (DCGM), DCGM Exporter, and the DRA validator. + - Supports multiple device injection strategies. + ``ClusterPolicy`` manages NVIDIA Container Toolkit, GPU Feature Discovery, MIG Manager, the sandbox device plugin, Kata Containers, KubeVirt, and NVIDIA vGPU Manager. + + * - Kubernetes Scheduling + - Uses ``ResourceClaim`` objects and DRA scheduler integration. + Kubernetes does not support preemption for DRA resources. + - Uses Kubernetes extended resource scheduling and supports Kubernetes Pod priority and preemption. + +DRA Driver Feature Maturity +=========================== + +GPU Operator support for deploying DRA components and the maturity of capabilities in the upstream DRA driver are separate considerations. +Upstream describes ComputeDomains as *officially supported* rather than *generally available (GA)*. +For this comparison, the table treats the terms as equivalent. +``GPUCluster`` always enables GPU allocation, can disable ComputeDomains, and passes ``draDriver.featureGates`` values to the DRA driver. + +The following table groups the DRA driver capabilities by maturity: + +.. list-table:: DRA Driver Capability Maturity + :header-rows: 1 + :widths: 34 22 18 26 + + * - Capability + - Maturity + - Default + - Operator Control + + * - ComputeDomains + - Officially supported (GA) + - Enabled + - ``draDriver.computeDomains.enabled`` + + * - Full GPU and existing MIG device allocation + - Not officially supported + - Enabled + - Always enabled + + * - ``ComputeDomainCliques``, ``CrashOnNVLinkFabricErrors``, ``IMEXDaemonsWithDNSNames`` + - Beta + - Enabled + - ``draDriver.featureGates`` + + * - ``DeviceMetadata``, ``DynamicMIG``, ``MPSSupport``, ``NVMLDeviceHealthCheck``, ``PassthroughSupport``, ``TimeSlicingSettings`` + - Alpha + - Disabled + - ``draDriver.featureGates`` + +Refer to the `DRA driver v${dra_version} overview `__ +and `feature-gate definitions `__ +for the authoritative support status, maturity, and default values. + +DRA Driver Limitations +====================== + +Consider the following limitations before selecting DRA driver v${dra_version}: + +* ComputeDomains are officially supported, but full GPU and MIG allocation are not yet officially supported upstream. +* ``NVMLDeviceHealthCheck`` is alpha and disabled by default. + The gRPC health probe reports kubelet plugin availability, and DCGM provides telemetry. + Neither provides DRA allocation health status. +* This release does not provide scheduler-accounted capacity sharing among independent ``ResourceClaim`` objects. + Workloads must share one claim instead of requesting independent replicas. +* Alpha feature gates have compatibility constraints. + ``DynamicMIG`` conflicts with ``PassthroughSupport``, ``NVMLDeviceHealthCheck``, and ``MPSSupport``. + ``PassthroughSupport`` conflicts with ``NVMLDeviceHealthCheck``, and ``DeviceMetadata`` requires ``PassthroughSupport``. +* Kubernetes does not support preemption for DRA resources. + Injecting DRA-allocated devices requires a preconfigured CDI-compatible container runtime. + The managed ``gpu.nvidia.com`` ``DeviceClass`` does not set ``spec.extendedResourceName`` to ``nvidia.com/gpu``. + ******** Overview ******** -The GPU Operator manages the DRA enablement stack through the ``nvidia.com/v1alpha1`` ``GPUCluster`` custom resource. +The GPU Operator manages DRA components through the ``nvidia.com/v1alpha1`` ``GPUCluster`` custom resource. ``GPUCluster`` is a cluster-scoped singleton and its name must be ``gpu-cluster``. The Helm chart creates this resource when you set ``gpuCluster.deployCR=true``. A cluster can have either a ``GPUCluster`` or a ``ClusterPolicy`` resource, but not both. -``ClusterPolicy`` manages the enablement stack based on the NVIDIA Kubernetes Device Plugin, -whereas ``GPUCluster`` manages the stack based on DRA. -Mixed Device Plugin and DRA stacks are not supported. +The GPU Operator uses ``ClusterPolicy`` to manage components for Device Plugin-based allocation +and ``GPUCluster`` to manage components for DRA-based allocation. -The ``GPUCluster`` stack includes the following components: +For ``GPUCluster``, the GPU Operator manages the following components: * The DRA driver GPU capability for ``gpu.nvidia.com``, ``mig.nvidia.com``, and ``vfio.gpu.nvidia.com`` devices. This capability is always enabled. @@ -61,8 +191,8 @@ The ``GPUCluster`` resource does not manage the NVIDIA GPU driver. Use an ``NVIDIADriver`` resource to install a containerized driver or use a driver that is pre-installed on the host. The Operator automatically assigns the ``nvidia.com/gpu.deploy.*`` labels that control operand placement. -Limitations -=========== +GPUCluster Limitations +====================== * Use this workflow for a new installation. An in-place migration from a standalone DRA driver Helm release or from ``ClusterPolicy`` to ``GPUCluster`` is not supported. @@ -89,9 +219,11 @@ In addition to ensuring that your GPUs and cluster align with the :ref:`GPU Oper .. note:: - To use traditional extended resource requests such as ``nvidia.com/gpu`` with the DRA driver, - enable the `DRAExtendedResource `_ feature gate. + To use an extended-resource request with the DRA driver, enable the + `DRAExtendedResource `_ feature gate. This feature gate is enabled by default in Kubernetes v1.36.0 and later. + Request ``deviceclass.resource.kubernetes.io/gpu.nvidia.com`` to use the managed ``gpu.nvidia.com`` ``DeviceClass``. + The legacy name ``nvidia.com/gpu`` requires a separate ``DeviceClass`` that sets ``spec.extendedResourceName`` to that value. .. _computedomain-prereqs: @@ -164,9 +296,9 @@ To install GPU allocation without ComputeDomain support, add the following optio Do not install a separate ``dra-driver-nvidia-gpu`` Helm release for this managed workflow. -*********************** -Configure the DRA Stack -*********************** +************************ +Configure DRA Components +************************ The GPU Operator Helm values render the specification of the ``gpu-cluster`` resource. The following settings provide the primary configuration surface: @@ -312,7 +444,7 @@ Additional validation procedures are available in the upstream DRA Driver docume Telemetry ********* -DCGM Exporter is enabled by default for the ``GPUCluster`` stack and uses its embedded host engine. +DCGM Exporter is enabled by default with ``GPUCluster`` and uses its embedded host engine. Set ``dcgm.enabled=true`` to deploy the standalone ``nvidia-dcgm-dra`` host engine instead. When either ``dcgmExporter.enablePodLabels`` or ``dcgmExporter.enablePodUID`` is enabled, From 2c23c2f2436c363a1b36b57e8158157cce165b91 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 18 Aug 2026 11:06:47 -0400 Subject: [PATCH 3/5] docs: Review from Babak Signed-off-by: Mike McKiernan --- gpu-operator/dra-intro-install.rst | 47 +++++++++---------- gpu-operator/getting-started.rst | 3 +- gpu-operator/gpu-driver-configuration.rst | 2 +- .../install-gpu-operator-air-gapped.rst | 2 +- 4 files changed, 25 insertions(+), 29 deletions(-) diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index c80dc4976..64913815b 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -20,8 +20,7 @@ DRA Driver for NVIDIA GPUs ########################## -Dynamic Resource Allocation (DRA) is a Kubernetes API for flexibly requesting, -configuring, and sharing specialized devices such as GPUs. +Dynamic Resource Allocation (DRA) is a Kubernetes API for flexibly requesting, configuring, and sharing specialized devices such as GPUs. This page describes how to use the GPU Operator to install and manage DRA Driver for NVIDIA GPUs v${dra_version}. Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the following documentation: @@ -31,14 +30,16 @@ Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the follo .. important:: - GPU Operator management of DRA is available as a technology preview. - Technology preview features are not supported in production environments and are not functionally complete. + GPU Operator supports the DRA Driver as a more recent alternative to the Device Plugin for Kubernetes. + However, some features of the DRA driver are alpha maturity and not fully supported. + The driver does not provide feature parity with the device plugin. + Refer to the following section regarding the features that are not yet implemented or fully supported before selecting the driver for production use. ********************************* Comparison: DRA and Device Plugin ********************************* -The DRA Driver for NVIDIA GPUs and the NVIDIA Kubernetes Device Plugin provide alternative mechanisms for allocating NVIDIA GPU resources. +The DRA Driver for NVIDIA GPUs and the NVIDIA Device Plugin for Kubernetes provide alternative mechanisms for allocating NVIDIA GPU resources. The mechanisms do not provide feature parity. A cluster can have either a ``GPUCluster`` resource for DRA or a ``ClusterPolicy`` resource for the Device Plugin, but not both. @@ -70,19 +71,22 @@ The following table compares the GPU allocation capabilities of the two mechanis - Device Plugin with ``ClusterPolicy`` * - API and Device Selection - - Uses DRA API objects for structured device selection and per-claim configuration. + - Uses DRA API to select GPUs by attributes---model, architecture, memory, compute capability, UUID, addressing mode, PCI/topology information---using CEL expressions. + + With Kubernetes v1.36, the driver also supports the extended resource names for backward compatibility. - Uses Kubernetes extended resource names and counts. - Configuration applies at the node or cluster level. + Node labeling or other external mechanisms required for structured selection. * - Full GPU and MIG Device Allocation - - Allocates full GPU devices and existing MIG devices. + - Allocates full GPU devices and preconfigured MIG devices. Alpha ``DynamicMIG`` creates MIG devices for a claim. - Allocates full GPU devices or preconfigured MIG devices. MIG Manager configures MIG geometry before allocation. * - GPU Sharing - - Alpha ``TimeSlicingSettings`` and ``MPSSupport`` configure sharing through DRA claims. - - Advertises time-slicing replicas and experimental MPS replicas as independent extended resources. + - Supports time-slicing through a shared DRA claim. Custom time-slice intervals require the Alpha ``TimeSlicingSettings`` feature gate. + MPS requires the Alpha ``MPSSupport`` gate. + - Advertises time-slicing replicas and experimental MPS replicas for all GPUs on a node. * - Multi-Node NVLink - Provides the ComputeDomain API and controller for coordinated workloads. @@ -100,18 +104,12 @@ The following table compares the GPU allocation capabilities of the two mechanis ``ClusterPolicy`` manages NVIDIA Container Toolkit, GPU Feature Discovery, MIG Manager, the sandbox device plugin, Kata Containers, KubeVirt, and NVIDIA vGPU Manager. * - Kubernetes Scheduling - - Uses ``ResourceClaim`` objects and DRA scheduler integration. - Kubernetes does not support preemption for DRA resources. - - Uses Kubernetes extended resource scheduling and supports Kubernetes Pod priority and preemption. + - Allocation through a ``ResourceClaim``, which can be referenced by multiple containers or Pods. + - Allocation through the resource limits specified for the container. DRA Driver Feature Maturity =========================== -GPU Operator support for deploying DRA components and the maturity of capabilities in the upstream DRA driver are separate considerations. -Upstream describes ComputeDomains as *officially supported* rather than *generally available (GA)*. -For this comparison, the table treats the terms as equivalent. -``GPUCluster`` always enables GPU allocation, can disable ComputeDomains, and passes ``draDriver.featureGates`` values to the DRA driver. - The following table groups the DRA driver capabilities by maturity: .. list-table:: DRA Driver Capability Maturity @@ -124,17 +122,17 @@ The following table groups the DRA driver capabilities by maturity: - Operator Control * - ComputeDomains - - Officially supported (GA) + - Generally Available - Enabled - ``draDriver.computeDomains.enabled`` * - Full GPU and existing MIG device allocation - - Not officially supported + - Generally Available - Enabled - Always enabled * - ``ComputeDomainCliques``, ``CrashOnNVLinkFabricErrors``, ``IMEXDaemonsWithDNSNames`` - - Beta + - Generally Available - Enabled - ``draDriver.featureGates`` @@ -152,7 +150,6 @@ DRA Driver Limitations Consider the following limitations before selecting DRA driver v${dra_version}: -* ComputeDomains are officially supported, but full GPU and MIG allocation are not yet officially supported upstream. * ``NVMLDeviceHealthCheck`` is alpha and disabled by default. The gRPC health probe reports kubelet plugin availability, and DCGM provides telemetry. Neither provides DRA allocation health status. @@ -169,7 +166,7 @@ Consider the following limitations before selecting DRA driver v${dra_version}: Overview ******** -The GPU Operator manages DRA components through the ``nvidia.com/v1alpha1`` ``GPUCluster`` custom resource. +The GPU Operator manages DRA components through the ``GPUCluster`` custom resource. ``GPUCluster`` is a cluster-scoped singleton and its name must be ``gpu-cluster``. The Helm chart creates this resource when you set ``gpuCluster.deployCR=true``. @@ -180,7 +177,7 @@ and ``GPUCluster`` to manage components for DRA-based allocation. For ``GPUCluster``, the GPU Operator manages the following components: * The DRA driver GPU capability for ``gpu.nvidia.com``, ``mig.nvidia.com``, and ``vfio.gpu.nvidia.com`` devices. - This capability is always enabled. + This capability is always enabled, although VFIO passthrough is unusable without the Alpha ``PassthroughSupport`` feature gate which is off by default. * The ComputeDomain controller and kubelet plugin for Multi-Node NVLink (MNNVL). ComputeDomain support is enabled by default and can be disabled. * A DRA validator that allocates a GPU by using a ``ResourceClaim`` and verifies that the device is usable. @@ -197,7 +194,7 @@ GPUCluster Limitations * Use this workflow for a new installation. An in-place migration from a standalone DRA driver Helm release or from ``ClusterPolicy`` to ``GPUCluster`` is not supported. * ``GPUCluster`` does not expose the controller affinity, priority class, toleration, - and kubelet-plugin node selector overrides that were used by the previous standalone DRA driver procedure for Google Kubernetes Engine. + and kubelet-plugin node selector overrides that were used by the previous standalone DRA driver procedure for Google Kubernetes Engine (GKE). This page does not provide a managed DRA installation procedure for GKE. ************* diff --git a/gpu-operator/getting-started.rst b/gpu-operator/getting-started.rst index 98df7f922..538348bcb 100644 --- a/gpu-operator/getting-started.rst +++ b/gpu-operator/getting-started.rst @@ -480,7 +480,6 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. * - ``draDriver.computeDomains.enabled`` - When set to ``true``, the ``GPUCluster`` stack deploys the ComputeDomain controller and kubelet plugin. - GPU allocation through DRA is always enabled when a ``GPUCluster`` is deployed. Refer to :doc:`DRA Driver for NVIDIA GPUs ` for more information. - ``true`` @@ -489,7 +488,7 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - ``v${dra_version}`` * - ``gpuCluster.deployCR`` - - When set to ``true``, the chart creates the technology preview ``GPUCluster`` resource for the DRA enablement stack. + - When set to ``true``, the chart creates the ``GPUCluster`` resource for the DRA enablement stack. Set ``clusterPolicy.deployCR=false`` and refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete installation procedure. - ``false`` diff --git a/gpu-operator/gpu-driver-configuration.rst b/gpu-operator/gpu-driver-configuration.rst index 46f3f85df..b799732f3 100644 --- a/gpu-operator/gpu-driver-configuration.rst +++ b/gpu-operator/gpu-driver-configuration.rst @@ -30,7 +30,7 @@ You can create one or more instances of an NVIDIA driver (``NVIDIADriver``) cust to specify the NVIDIA GPU driver type and driver version to configure on specific nodes. You can specify labels in the node selector field to control which NVIDIA driver configuration is applied to specific nodes. -The technology preview ``GPUCluster`` enablement stack uses ``NVIDIADriver`` resources for an Operator-managed driver. +The ``GPUCluster`` enablement stack uses ``NVIDIADriver`` resources for an Operator-managed driver. ``GPUCluster`` manages the DRA operands and waits for the matching NVIDIA driver deployment to become ready. Refer to :doc:`DRA Driver for NVIDIA GPUs ` for the complete ``GPUCluster`` installation procedure. diff --git a/gpu-operator/install-gpu-operator-air-gapped.rst b/gpu-operator/install-gpu-operator-air-gapped.rst index f4180faff..48c8c43d0 100644 --- a/gpu-operator/install-gpu-operator-air-gapped.rst +++ b/gpu-operator/install-gpu-operator-air-gapped.rst @@ -240,7 +240,7 @@ Sample of ``values.yaml`` for GPU Operator v1.9.0: DRA Driver Image ================ -If you deploy the technology preview ``GPUCluster`` custom resource definition, mirror the DRA Driver for NVIDIA GPUs image in addition to the other GPU Operator images: +If you deploy the ``GPUCluster`` custom resource definition, mirror the DRA Driver for NVIDIA GPUs image in addition to the other GPU Operator images: .. code-block:: console From d5a2bfe5964a4c4e4528e3c7a1a56e15c4c9ffc3 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 18 Aug 2026 14:26:20 -0400 Subject: [PATCH 4/5] docs: feedback from Francis Signed-off-by: Mike McKiernan --- gpu-operator/dra-intro-install.rst | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index 64913815b..ccba58182 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -28,19 +28,14 @@ Before using the DRA Driver for NVIDIA GPUs, familiarize yourself with the follo * `Dynamic Resource Allocation `_ in the Kubernetes documentation. * `DRA Driver for NVIDIA GPUs documentation `__. -.. important:: - - GPU Operator supports the DRA Driver as a more recent alternative to the Device Plugin for Kubernetes. - However, some features of the DRA driver are alpha maturity and not fully supported. - The driver does not provide feature parity with the device plugin. - Refer to the following section regarding the features that are not yet implemented or fully supported before selecting the driver for production use. - ********************************* Comparison: DRA and Device Plugin ********************************* -The DRA Driver for NVIDIA GPUs and the NVIDIA Device Plugin for Kubernetes provide alternative mechanisms for allocating NVIDIA GPU resources. -The mechanisms do not provide feature parity. +GPU Operator supports the DRA Driver as a more recent alternative to the NVIDIA Device Plugin for Kubernetes. +However, some features of the DRA driver are alpha maturity and not fully supported. +The driver does not provide feature parity with the device plugin. + A cluster can have either a ``GPUCluster`` resource for DRA or a ``ClusterPolicy`` resource for the Device Plugin, but not both. Use DRA for workloads that have the following requirements: @@ -117,7 +112,7 @@ The following table groups the DRA driver capabilities by maturity: :widths: 34 22 18 26 * - Capability - - Maturity + - Support Status - Default - Operator Control @@ -432,10 +427,19 @@ During reconciliation, the STATUS column progresses from empty to `notReady` to The ComputeDomain DeviceClasses are present only when ComputeDomain support is enabled. -Additional validation procedures are available in the upstream DRA Driver documentation: +************************* +Running a Sample Workload +************************* + +Refer to `Request full GPUs `__ +in the DRA driver documentation for information about the following workload-related tasks: -* `Run a sample ComputeDomain workload `__. -* `Run a sample GPU allocation workload `__. +* Request any GPU +* Request multiple GPUs in one pod +* Share a GPU across containers in a pod +* Select a GPU by product name +* Select a GPU by memory size +* Combine attribute and capacity selectors ********* Telemetry From 8a603aed7cd215906bd6bdddab69e7f849ff0f9a Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Fri, 21 Aug 2026 14:35:58 -0400 Subject: [PATCH 5/5] docs: Feedback from Shiva Signed-off-by: Mike McKiernan --- gpu-operator/dra-intro-install.rst | 32 +++++++++++------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/gpu-operator/dra-intro-install.rst b/gpu-operator/dra-intro-install.rst index ccba58182..cb01ae13d 100644 --- a/gpu-operator/dra-intro-install.rst +++ b/gpu-operator/dra-intro-install.rst @@ -33,25 +33,10 @@ Comparison: DRA and Device Plugin ********************************* GPU Operator supports the DRA Driver as a more recent alternative to the NVIDIA Device Plugin for Kubernetes. -However, some features of the DRA driver are alpha maturity and not fully supported. -The driver does not provide feature parity with the device plugin. +However, some features of the DRA driver have alpha support status. A cluster can have either a ``GPUCluster`` resource for DRA or a ``ClusterPolicy`` resource for the Device Plugin, but not both. -Use DRA for workloads that have the following requirements: - -* Coordinate Multi-Node NVLink workloads by using ComputeDomains. -* Use attribute-based GPU selection or allocation-specific device configuration instead of node-wide configuration. -* Configure dynamic NVIDIA Multi-Instance GPU (MIG), CUDA Multi-Process Service (MPS), - CUDA time-slicing, or Virtual Function I/O (VFIO) passthrough for individual allocations. - -Use the Device Plugin for workloads that have the following requirements: - -* Request ``nvidia.com/gpu`` or MIG extended resources from existing Pod specifications and tools. -* Schedule CUDA time-slicing or MPS replicas as independent extended resources. -* Use the broader set of components managed through ``ClusterPolicy``, Device Plugin health reporting, - or Kubernetes Pod priority and preemption. - Capability Comparison ===================== @@ -79,7 +64,9 @@ The following table compares the GPU allocation capabilities of the two mechanis MIG Manager configures MIG geometry before allocation. * - GPU Sharing - - Supports time-slicing through a shared DRA claim. Custom time-slice intervals require the Alpha ``TimeSlicingSettings`` feature gate. + - Alpha ``ConsumableShares`` feature gate implements Kubernetes consumable capacity for scheduler-accounted sharing among independent ``ResourceClaim`` objects for full GPUs and MIG devices. + The driver also supports time-slicing through a shared DRA claim. + Custom time-slice intervals require the Alpha ``TimeSlicingSettings`` feature gate. MPS requires the Alpha ``MPSSupport`` gate. - Advertises time-slicing replicas and experimental MPS replicas for all GPUs on a node. @@ -131,7 +118,7 @@ The following table groups the DRA driver capabilities by maturity: - Enabled - ``draDriver.featureGates`` - * - ``DeviceMetadata``, ``DynamicMIG``, ``MPSSupport``, ``NVMLDeviceHealthCheck``, ``PassthroughSupport``, ``TimeSlicingSettings`` + * - ``ConsumableShares``, ``DeviceMetadata``, ``DynamicMIG``, ``MPSSupport``, ``NVMLDeviceHealthCheck``, ``PassthroughSupport``, ``TimeSlicingSettings`` - Alpha - Disabled - ``draDriver.featureGates`` @@ -148,8 +135,13 @@ Consider the following limitations before selecting DRA driver v${dra_version}: * ``NVMLDeviceHealthCheck`` is alpha and disabled by default. The gRPC health probe reports kubelet plugin availability, and DCGM provides telemetry. Neither provides DRA allocation health status. -* This release does not provide scheduler-accounted capacity sharing among independent ``ResourceClaim`` objects. - Workloads must share one claim instead of requesting independent replicas. +* Consumable capacity is available through the alpha ``ConsumableShares`` feature gate, which is disabled by default. + When enabled, the feature provides scheduler-accounted capacity sharing among independent ``ResourceClaim`` objects for full GPUs and MIG devices. + Kubernetes v1.34 and v1.35 require enabling the ``DRAConsumableCapacity`` feature gate. + Kubernetes v1.36 and later enable the feature by default. + You must also set the ``CONSUMABLE_SHARES`` environment variable for the DRA GPU kubelet plugin to ``memory``, ``unlimited``, or a positive integer. + Set the environment variable through ``draDriver.gpus.kubeletPlugin.env``. + MPS cannot be used when consumable capacity is configured through ``ConsumableShares``. * Alpha feature gates have compatibility constraints. ``DynamicMIG`` conflicts with ``PassthroughSupport``, ``NVMLDeviceHealthCheck``, and ``MPSSupport``. ``PassthroughSupport`` conflicts with ``NVMLDeviceHealthCheck``, and ``DeviceMetadata`` requires ``PassthroughSupport``.