Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
624 changes: 432 additions & 192 deletions gpu-operator/dra-intro-install.rst

Large diffs are not rendered by default.

24 changes: 23 additions & 1 deletion gpu-operator/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,11 @@ To view all the options, run ``helm show values nvidia/gpu-operator``.
- When set to ``true``, the Operator deploys NVIDIA Confidential Computing Manager for Kubernetes.
- ``false``

* - ``clusterPolicy.deployCR``
- When set to ``true``, the chart creates the ``ClusterPolicy`` resource for the NVIDIA Kubernetes Device Plugin enablement stack.
Set this value to ``false`` when using ``gpuCluster.deployCR=true``.
- ``true``

* - ``cdi.enabled``
- When set to ``true`` (default), the Container Device Interface (CDI) will be used for
injecting GPUs into workload containers.
Expand Down Expand Up @@ -415,13 +420,15 @@ To view all the options, run ``helm show values nvidia/gpu-operator``.
- When set to ``true``, Kubernetes pod labels are added as Prometheus label dimensions on the GPU metrics.
Enabling this option causes the Operator to provision a cluster-scoped ClusterRole and ClusterRoleBinding
(``nvidia-dcgm-exporter-read-pods``) that grants the DCGM Exporter service account ``get``, ``list``, and ``watch`` access to pods.
For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects.
Use ``dcgmExporter.podLabelAllowlistRegex`` to limit which labels are emitted.
- ``false``

* - ``dcgmExporter.enablePodUID``
- When set to ``true``, the Kubernetes pod UID is added as a Prometheus label dimension on the GPU metrics.
Like ``dcgmExporter.enablePodLabels``, this provisions a cluster-scoped ClusterRole and ClusterRoleBinding that grants the DCGM Exporter
service account ``get``, ``list``, and ``watch`` access to pods.
For a ``GPUCluster`` installation, enabling this option also enables DRA attribution and grants read access to ``ResourceSlice`` objects.
- ``false``

* - ``dcgmExporter.podLabelAllowlistRegex``
Expand Down Expand Up @@ -470,6 +477,20 @@ To view all the options, run ``helm show values nvidia/gpu-operator``.
custom driver images.
- ``nvcr.io/nvidia``

* - ``draDriver.computeDomains.enabled``
- When set to ``true``, the ``GPUCluster`` stack deploys the ComputeDomain controller and kubelet plugin.
Refer to :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>` for more information.
- ``true``

* - ``draDriver.version``
- Version of the DRA Driver for NVIDIA GPUs deployed by the ``GPUCluster`` stack.
- ``v${dra_version}``

* - ``gpuCluster.deployCR``
- When set to ``true``, the chart creates the ``GPUCluster`` resource for the DRA enablement stack.
Set ``clusterPolicy.deployCR=false`` and refer to :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>` for the complete installation procedure.
- ``false``

* - ``driver.rdma.enabled``
- Controls whether the driver daemon set builds and loads the legacy ``nvidia-peermem`` kernel module.

Expand Down Expand Up @@ -891,6 +912,7 @@ After verifying the installation, you can configure the GPU Operator for your wo
- :doc:`gpu-sharing` — Share a single GPU across multiple pods using time-slicing or MPS.
- :doc:`gpu-operator-mig` — Configure Multi-Instance GPU (MIG) partitioning on supported GPUs.
- :doc:`gpu-operator-rdma` — Enable GPUDirect RDMA for high-performance networking.
- :doc:`dra-intro-install` — Allocate GPUs by using Kubernetes Dynamic Resource Allocation (DRA).
- :doc:`gpu-driver-configuration` — Use the NVIDIA GPU Driver Custom Resource Definition to manage drivers per node.
- :doc:`precompiled-drivers` — Speed up driver deployments with precompiled kernel modules.
- :doc:`cdi` — Learn about Container Device Interface (CDI) and NRI Plugin mode.
- :doc:`cdi` — Learn about Container Device Interface (CDI) and NRI Plugin mode.
10 changes: 10 additions & 0 deletions gpu-operator/gpu-driver-configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ You can create one or more instances of an NVIDIA driver (``NVIDIADriver``) cust
to specify the NVIDIA GPU driver type and driver version to configure on specific nodes.
You can specify labels in the node selector field to control which NVIDIA driver configuration is applied to specific nodes.

The ``GPUCluster`` enablement stack uses ``NVIDIADriver`` resources for an Operator-managed driver.
``GPUCluster`` manages the DRA operands and waits for the matching NVIDIA driver deployment to become ready.
Refer to :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>` for the complete ``GPUCluster`` installation procedure.


Limitations
===========
Expand Down Expand Up @@ -260,6 +264,12 @@ Installing the NVIDIA GPU Operator

Perform the following steps to install the GPU Operator and use the NVIDIA driver custom resources.

.. note::

To use an ``NVIDIADriver`` resource with the ``GPUCluster`` DRA stack,
follow the installation procedure in :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>`.
That procedure configures the chart to create ``GPUCluster`` instead of ``ClusterPolicy``.

#. Optional: If you want to run more than one driver type or version in the cluster,
label the worker nodes to identify the driver type and version to install on each node:

Expand Down
27 changes: 25 additions & 2 deletions gpu-operator/gpu-driver-upgrades.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ The GPU Operator supports several methods for managing and automating this drive
The GPU Operator only manages the lifecycle of containerized drivers.
Drivers which are pre-installed on the host are not managed by the GPU Operator.

For the ``GPUCluster`` DRA stack, the Operator treats pods with an allocated
``gpu.nvidia.com`` ResourceClaim as GPU workloads during a driver upgrade.
The configured pod deletion policy applies to these workloads before the driver is reloaded.
The DRA kubelet plugin remains available while terminating pods unprepare their claims.


************************************
Upgrades with the Upgrade Controller
Expand All @@ -56,14 +61,24 @@ The controller automates the upgrade process and generates metrics and events so

.. rubric:: Procedure

1. Upgrade the driver by changing the ``driver.version`` value in the cluster policy:
1. Upgrade the driver by changing the version in the resource that manages the driver.

For a ``ClusterPolicy``-managed driver, change ``driver.version``:

.. code-block:: console

$ kubectl patch clusterpolicies.nvidia.com/cluster-policy \
--type='json' \
-p='[{"op": "replace", "path": "/spec/driver/version", "value":"580.95.05"}]'

For an ``NVIDIADriver``-managed driver, such as the default driver used with ``GPUCluster``, change ``spec.version``:

.. code-block:: console

$ kubectl patch nvidiadrivers.nvidia.com/default \
--type='json' \
-p='[{"op": "replace", "path": "/spec/version", "value":"580.95.05"}]'


If you are using Openshift, you must update the ``driver.version``, ``driver.repository`` and ``driver.image`` values in the cluster policy.

Expand All @@ -73,6 +88,15 @@ The controller automates the upgrade process and generates metrics and events so
--type='json' \
-p='[{"op": "replace", "path": "/spec/driver/version", "value":"580.95.05"},{"op": "replace", "path": "/spec/driver/repository", "value":"nvcr.io/nvidia"},{"op": "replace", "path": "/spec/driver/image", "value":"driver"}]'

For an ``NVIDIADriver``-managed driver on OpenShift, including a driver used with ``GPUCluster``,
update ``spec.version``, ``spec.repository``, and ``spec.image`` on the ``NVIDIADriver`` resource instead:

.. code-block:: console

$ kubectl patch nvidiadrivers.nvidia.com/default \
--type='json' \
-p='[{"op": "replace", "path": "/spec/version", "value":"580.95.05"},{"op": "replace", "path": "/spec/repository", "value":"nvcr.io/nvidia"},{"op": "replace", "path": "/spec/image", "value":"driver"}]'

2. (Optional) For each node, monitor the upgrade status:

.. code-block:: console
Expand Down Expand Up @@ -361,4 +385,3 @@ GPU pod eviction and node drain behavior.
With ``OnDelete`` update strategy, a new driver pod with the updated spec will only get deployed on a node once the old driver pod is manually deleted.
Thus, admins can control when to rollout spec updates to driver pods on any given node.
For more information on DaemonSet update strategies, refer to the `Kubernetes documentation <https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/#daemonset-update-strategy>`_.

23 changes: 23 additions & 0 deletions gpu-operator/install-gpu-operator-air-gapped.rst
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,29 @@ Sample of ``values.yaml`` for GPU Operator v1.9.0:
imagePullSecrets: []


DRA Driver Image
================

If you deploy the ``GPUCluster`` custom resource definition, mirror the DRA Driver for NVIDIA GPUs image in addition to the other GPU Operator images:

.. code-block:: console

$ docker pull registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v${dra_version}

Tag and push this image to the local registry, then configure its location in ``values.yaml``:

.. code-block:: yaml

draDriver:
repository: <repo.example.com:port>/<local-path>
image: dra-driver-nvidia-gpu
version: v${dra_version}
imagePullSecrets: []

Use the remaining ``GPUCluster`` installation settings from :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>`.
The ``GPUCluster`` and ComputeDomain custom resource definitions are packaged in the GPU Operator chart and do not require a separate download during installation.


************************
Local Package Repository
************************
Expand Down
37 changes: 36 additions & 1 deletion gpu-operator/uninstall.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,28 @@ Uninstalling the GPU Operator

Perform the following steps to uninstall the Operator.

#. If a ``GPUCluster`` resource exists, delete user workloads and user-created ResourceClaims for NVIDIA GPUs.
Wait for the workload pods to terminate and the claims to be unprepared.

.. code-block:: console

$ kubectl get gpucluster gpu-cluster
$ kubectl get resourceclaim --all-namespaces

The Helm uninstall command later in this procedure runs a ``pre-delete`` hook that deletes ``gpu-cluster``.
The hook waits for the GPUCluster finalizer to remove Operator-managed ResourceClaim consumers before Helm removes the Operator.

#. Optional: List and delete NVIDIA driver custom resources.

If you use ``GPUCluster``, skip this optional step unless you first delete ``gpu-cluster`` and wait for its finalizer to complete:

.. code-block:: console

$ kubectl delete gpucluster gpu-cluster
$ kubectl wait --for=delete gpucluster/gpu-cluster --timeout=5m

The chart-managed default ``NVIDIADriver`` resource does not need to be deleted separately.

.. code-block:: console

$ kubectl get nvidiadrivers
Expand Down Expand Up @@ -67,16 +87,27 @@ By default, Helm does not `support deleting existing CRDs <https://helm.sh/docs/
when you delete the chart.
As a result, the ``clusterpolicy`` CRD and ``nvidiadrivers`` CRD will still remain, by default.

The ``gpuclusters.nvidia.com``, ``computedomains.resource.nvidia.com``, and
``computedomaincliques.resource.nvidia.com`` CRDs also remain after a ``GPUCluster`` installation.

.. code-block:: console

$ kubectl get crd clusterpolicies.nvidia.com

To overcome this, the Operator uses a `post-delete hook <https://helm.sh/docs/topics/charts_hooks/#the-available-hooks>`__
To overcome this, the Operator uses a `pre-delete hook <https://helm.sh/docs/topics/charts_hooks/#the-available-hooks>`__
to perform the CRD cleanup.
The ``operator.cleanupCRD`` chart parameter is added to enable this hook.
This parameter is disabled by default.
You can enable the hook by specifying ``--set operator.cleanupCRD=true`` during install or upgrade to perform automatic CRD cleanup on chart deletion.

The cleanup hook removes the ``gpuclusters.nvidia.com`` CRD, but does not remove the two ComputeDomain CRDs.
After verifying that no ComputeDomain resources remain, you can remove those definitions manually:

.. code-block:: console

$ kubectl delete crd computedomains.resource.nvidia.com
$ kubectl delete crd computedomaincliques.resource.nvidia.com

Alternatively, you can delete the custom resource definition:

.. code-block:: console
Expand All @@ -95,3 +126,7 @@ Alternatively, you can delete the custom resource definition:
* Helm hooks used with the GPU Operator use the Operator image itself.
If the Operator image cannot be pulled successfully (either due to network error or an invalid NGC registry secret in case of NVAIE), hooks will fail.
In this case, delete the chart and specify the ``--no-hooks`` argument to avoid hanging on hook failures.

If a ``GPUCluster`` resource exists, do not specify ``--no-hooks``.
Doing so can remove the Operator before the GPUCluster finalizer completes ordered teardown of ResourceClaim-consuming operands.
First resolve the hook failure, or delete ``gpu-cluster`` and wait for it to disappear while the Operator is still running.
26 changes: 24 additions & 2 deletions gpu-operator/upgrade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ With this procedure, all existing GPU Operator resources are updated inline and

$ export RELEASE_TAG=${version}

#. Apply the custom resource definitions for the cluster policy and NVIDIA driver:
#. Apply the custom resource definitions for the cluster policy, NVIDIA driver, and GPU cluster:

.. code-block:: console

Expand All @@ -75,12 +75,28 @@ With this procedure, all existing GPU Operator resources are updated inline and
$ kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml

$ kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_gpuclusters.yaml

*Example Output*

.. code-block:: output

customresourcedefinition.apiextensions.k8s.io/clusterpolicies.nvidia.com configured
customresourcedefinition.apiextensions.k8s.io/nvidiadrivers.nvidia.com created
customresourcedefinition.apiextensions.k8s.io/gpuclusters.nvidia.com created

#. If you use :doc:`DRA Driver for NVIDIA GPUs <dra-intro-install>`, apply the ComputeDomain custom resource definitions:

.. code-block:: console

$ kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomains.yaml

$ kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/resource.nvidia.com_computedomaincliques.yaml

Apply both definitions before upgrading, including when ComputeDomain support is currently disabled.

#. Apply the custom resource definition for Node Feature Discovery:

Expand Down Expand Up @@ -143,6 +159,12 @@ Starting with GPU Operator v22.09, a ``pre-upgrade`` Helm `hook <https://helm.sh

Starting with GPU Operator v24.9.0, the upgrade CRD Helm hook is enabled by default and runs an upgrade CRD job when you upgrade using Helm.

.. important::

For a ``GPUCluster`` installation, manually apply the ``computedomains.resource.nvidia.com`` and
``computedomaincliques.resource.nvidia.com`` custom resource definitions by using the commands in Option 1 before you upgrade.
The automatic hook upgrades the ``GPUCluster`` custom resource definition but does not upgrade the ComputeDomain custom resource definitions.

#. Specify the Operator release tag in an environment variable:

.. code-block:: console
Expand Down Expand Up @@ -215,4 +237,4 @@ Refer to :ref:`GPU Driver Upgrades` for more information.
Using Operator Lifecycle Manager (OLM) in OpenShift
***************************************************

For upgrading the GPU Operator when running in OpenShift, refer to the official OpenShift documentation on `upgrading installed operators <https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/operators/administrator-tasks#olm-upgrading-operators>`_.
For upgrading the GPU Operator when running in OpenShift, refer to the official OpenShift documentation on `upgrading installed operators <https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/operators/administrator-tasks#olm-upgrading-operators>`_.