From 10cf2a4c5da51b02115a11f1781171f1a62a9eaf Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Fri, 31 Jul 2026 11:59:30 -0400 Subject: [PATCH 1/3] docs: CNT 1.20 updates Signed-off-by: Mike McKiernan --- container-toolkit/cdi-support.md | 81 ++++++++++++++++++++++++- container-toolkit/docker-specialized.md | 49 ++++++++++++++- container-toolkit/index.md | 1 + container-toolkit/install-guide.md | 43 +++++++++++++ container-toolkit/nri-support.md | 81 +++++++++++++++++++++++++ container-toolkit/release-notes.md | 56 +++++++++++++++++ container-toolkit/versions1.json | 48 ++------------- repo.toml | 4 +- 8 files changed, 314 insertions(+), 49 deletions(-) create mode 100644 container-toolkit/nri-support.md diff --git a/container-toolkit/cdi-support.md b/container-toolkit/cdi-support.md index 1e9dd2273..ccb278ae2 100644 --- a/container-toolkit/cdi-support.md +++ b/container-toolkit/cdi-support.md @@ -161,6 +161,59 @@ directly: $ sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml ``` +## JIT-CDI Mode + +Just-in-time CDI (JIT-CDI) mode generates an in-memory CDI specification for the +NVIDIA devices that a container requests. The NVIDIA Container Runtime uses this +specification to update the container configuration. JIT-CDI mode does not write +a persistent specification to `/var/run/cdi` or `/etc/cdi`. + +When `nvidia-container-runtime.mode` is set to `auto`, the runtime selects JIT-CDI +mode on systems that use the NVIDIA Management Library (NVML) or Windows +Subsystem for Linux 2 (WSL2). Native CDI-enabled runtimes still use the +persistent specifications described in +[Automatic CDI Specification Generation](#automatic-cdi-specification-generation). + +### Disabling Hooks in JIT-CDI Mode + +You can prevent JIT-CDI mode from adding specific hooks to an in-memory +specification. The supported hook names are `chmod`, `create-symlinks`, +`disable-device-node-modification`, `enable-cuda-compat`, +`update-application-profile`, and `update-ldcache`. The deprecated `chmod` hook +is disabled by default. Use `all` to disable every hook. + +The following command disables the application-profile and dynamic-linker cache +hooks. Separate multiple values with a colon: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks=update-application-profile:update-ldcache +``` + +The resulting configuration contains the following settings: + +```toml +[nvidia-container-runtime.modes.jit-cdi] +nvcdi-disable-hooks = ["update-application-profile", "update-ldcache"] +``` + +```{warning} +Disable only the hook that conflicts with your environment. Hooks configure +libraries, links, device behavior, and GPU visibility in the container. Disabling +a required hook can prevent an application from starting or can expose more GPUs +to EGL and Vulkan applications than the container requested. +``` + +For a persistent specification, pass `--disable-hook` once for each hook when you +[generate the CDI specification manually](#manual-cdi-specification-generation): + +```console +$ sudo nvidia-ctk cdi generate \ + --disable-hook update-application-profile \ + --disable-hook update-ldcache \ + --output=/var/run/cdi/nvidia.yaml +``` + ## Running a Workload with CDI Using CDI to inject NVIDIA devices can conflict with using the NVIDIA Container Runtime hook. @@ -191,6 +244,32 @@ $ podman run --rm \ The preceding sample command requests the full GPU with index 0 and the first MIG device on GPU 1. The output should show only the UUIDs of the requested devices. +### Graphics and OpenCL Workloads + +CDI specifications include the driver libraries and configuration files that +graphics and OpenCL applications require. When the NVIDIA OpenCL installable +client driver (ICD) file is present on the host, the specification mounts it at +`/etc/OpenCL/vendors/nvidia.icd` in the container. + +CDI specifications also include the `update-application-profile` hook. At +container creation, this hook writes +`/etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf`. The profile +limits EGL and Vulkan visibility to the physical GPUs that are available in the +container. As a result, applications do not enumerate unassigned physical GPUs. + +If the application-profile hook conflicts with an application, disable it only +for that workload or environment. For JIT-CDI mode, refer to +[Disabling Hooks in JIT-CDI Mode](#disabling-hooks-in-jit-cdi-mode). For a +persistent specification, use the `--disable-hook update-application-profile` +option described in [Manual CDI Specification Generation](#manual-cdi-specification-generation). + +### IMEX Channels + +Containers can request specific NVIDIA IMEX channels with the +`NVIDIA_IMEX_CHANNELS` environment variable. For supported values, an example, +and validation errors, refer to +[Requesting IMEX Channels](docker-specialized.md#requesting-imex-channels). + ## Using CDI with Non-CDI-Enabled Runtimes To support runtimes that do not natively support CDI, you can configure the NVIDIA Container Runtime in a `cdi` mode. @@ -237,4 +316,4 @@ $ docker run --rm -ti --runtime=nvidia \ - [Container Device Interface](https://github.com/cncf-tags/container-device-interface) (CDI) specification from the Container Device Interface repository on GitHub. - [How to configure CDI](https://github.com/cncf-tags/container-device-interface#how-to-configure-cdi) from the GitHub repository provides an overview of manual configuration for CRI-O, containerd, and Podman. - The NVIDIA Container Toolkit performs the configuration for you. \ No newline at end of file + The NVIDIA Container Toolkit performs the configuration for you. diff --git a/container-toolkit/docker-specialized.md b/container-toolkit/docker-specialized.md index 6b88c1c5e..465308ab9 100644 --- a/container-toolkit/docker-specialized.md +++ b/container-toolkit/docker-specialized.md @@ -111,6 +111,34 @@ The following examples show common usage: nvidia/cuda nvidia-smi ``` +(requesting-imex-channels)= + +### Requesting IMEX Channels + +Use the `NVIDIA_IMEX_CHANNELS` environment variable to request NVIDIA IMEX +channels for a container. Specify one or more numeric channel IDs as a +comma-separated list. + +The following command requests channels 0 and 1: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=all \ + -e NVIDIA_IMEX_CHANNELS=0,1 \ + +``` + +In CDI and JIT-CDI mode, each channel ID must meet both requirements: + +- The ID is in the range from 0 through 1,048,575. +- The corresponding `/dev/nvidia-caps-imex-channels/channel` device exists + on the host. + +If either requirement is not met, container creation fails with an error that +identifies the invalid or missing channel. Inspect +`/dev/nvidia-caps-imex-channels/` on the host and request only the channel IDs +that are present. + ### Driver Capabilities The `NVIDIA_DRIVER_CAPABILITIES` variable controls which driver libraries and binaries are mounted inside the container. @@ -146,13 +174,14 @@ The following table describes the supported driver capabilities: - Description * - ``compute`` - - required for CUDA and OpenCL applications. + - Required for CUDA and OpenCL applications. When present on the host, + the NVIDIA OpenCL ICD file is also available in the container. * - ``compat32`` - required for running 32-bit applications. * - ``graphics`` - - required for running OpenGL and Vulkan applications. + - Required for running OpenGL, EGL, and Vulkan applications. * - ``utility`` - required for using ``nvidia-smi`` and NVML. @@ -178,6 +207,22 @@ For example, to allow usage of CUDA and NVML, specify the `compute` and `utility > nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi > ``` +To run an OpenGL, EGL, or Vulkan application on a selected GPU, include the +`graphics` capability. The following command makes GPU 0 and the graphics and +utility driver components available to the container: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=0 \ + -e NVIDIA_DRIVER_CAPABILITIES=graphics,utility \ + +``` + +When CDI or JIT-CDI injects the GPU, the toolkit also limits EGL and Vulkan +visibility to the physical GPUs assigned to the container. For more information +and exceptional compatibility settings, refer to +[Graphics and OpenCL Workloads](cdi-support.md#graphics-and-opencl-workloads). + ### Constraints The NVIDIA runtime also lets you define constraints on the configurations that the container supports. diff --git a/container-toolkit/index.md b/container-toolkit/index.md index 3cab94725..45253d4f2 100644 --- a/container-toolkit/index.md +++ b/container-toolkit/index.md @@ -20,6 +20,7 @@ release-notes.md arch-overview.md Container Device Interface +Node Resource Interface docker-specialized.md ``` diff --git a/container-toolkit/install-guide.md b/container-toolkit/install-guide.md index 56282dab4..81480f170 100644 --- a/container-toolkit/install-guide.md +++ b/container-toolkit/install-guide.md @@ -149,6 +149,49 @@ where `systemd` cgroup drivers are used that cause containers to lose access to * You installed a supported container engine (Docker, Containerd, CRI-O, Podman). * You installed the NVIDIA Container Toolkit. +### Managing `config.toml` + +The `nvidia-ctk config` command reads +`/etc/nvidia-container-runtime/config.toml` by default. Without `--in-place` or +`--output`, the command writes the updated configuration to standard output and +does not change the source file. + +Preview a change before you write it to the host: + +```console +$ sudo nvidia-ctk config \ + --set nvidia-container-runtime.log-level=debug +``` + +Review the output. Then, add `--in-place` to update the source file: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug +``` + +You can specify `--set` more than once. The following command configures a log +level and a log file in one update: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug \ + --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log +``` + +For a list setting, separate elements with a colon. The following command makes +`crun` the first low-level runtime candidate and retains `runc` as a fallback: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.runtimes=crun:runc +``` + +When the same key appears more than once, the last value takes effect. The +command preserves existing settings when you change a different setting. Review +the preview before every production update, especially when you use a custom +file with `--config-file`. + (setting-up-docker)= ### Configuring Docker diff --git a/container-toolkit/nri-support.md b/container-toolkit/nri-support.md new file mode 100644 index 000000000..d767d71b0 --- /dev/null +++ b/container-toolkit/nri-support.md @@ -0,0 +1,81 @@ +(nri-support)= + +# Support for Node Resource Interface + +## About the NRI Plugin + +The NVIDIA Container Toolkit container can run a Node Resource Interface (NRI) +plugin. The plugin connects to an NRI-enabled container runtime and adds +Container Device Interface (CDI) device requests to container configurations. +This path is useful when the runtime cannot process the original CDI request +directly. + +The plugin enforces different namespace rules for workload and management +devices: + +- The plugin can inject ordinary workload devices, such as `nvidia.com/gpu`, + into containers in any namespace. +- The plugin injects `management.nvidia.com/gpu` devices only into the toolkit + namespace or an additional namespace that an administrator explicitly + authorizes. + +## Enabling the NRI Plugin + +The container runtime must have NRI enabled, and the toolkit container must have +access to the runtime's NRI socket. Keep the toolkit installer running as a +daemon. The `--enable-nri-plugin` option is ignored when `--no-daemon` is set. + +Add the following options to the toolkit container entry point: + +```console +$ nvidia-ctk-installer \ + --enable-nri-plugin \ + --nri-namespace=nvidia-gpu-operator +``` + +The `--nri-namespace` value must match the Kubernetes namespace that contains +the toolkit pod. Use `--nri-socket` if the runtime exposes the NRI socket at a +nondefault path. + +You can configure the same settings with environment variables: + +```yaml +env: + - name: ENABLE_NRI_PLUGIN + value: "true" + - name: NRI_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace +``` + +## Authorizing Management Devices in Additional Namespaces + +Use `--nri-management-cdi-device-namespaces` to allow a namespace to receive +management CDI devices. You can repeat the option. The toolkit namespace is +always allowed and does not need to appear in the list. + +The following example authorizes management devices in two additional +namespaces: + +```console +$ nvidia-ctk-installer \ + --enable-nri-plugin \ + --nri-namespace=nvidia-gpu-operator \ + --nri-management-cdi-device-namespaces=gpu-monitoring \ + --nri-management-cdi-device-namespaces=gpu-diagnostics +``` + +For a container deployment, the equivalent environment variable accepts a +comma-separated list: + +```yaml +env: + - name: NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES + value: "gpu-monitoring,gpu-diagnostics" +``` + +Namespace matching is exact. Wildcards are not supported. Allow only namespaces +that contain trusted management workloads. A management CDI device can provide +broader access than an ordinary workload device, so follow the principle of +least privilege. diff --git a/container-toolkit/release-notes.md b/container-toolkit/release-notes.md index 29521bfe9..394207e85 100644 --- a/container-toolkit/release-notes.md +++ b/container-toolkit/release-notes.md @@ -8,6 +8,62 @@ This document describes the new features, improvements, fixes and known issues for the NVIDIA Container Toolkit. +## NVIDIA Container Toolkit 1.20.0 + +This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. + +### Fixes and Features + +- CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container. + Graphics applications no longer see unassigned host GPUs through these APIs. +- CUDA compatibility handling now uses `libcuda.so` ELF metadata whenever it is available. + This improves CUDA minor-version compatibility by selecting the container's compatibility libraries only when they are appropriate for the installed driver. +- Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version. + This fix avoids both missing required libraries and injecting libraries from another installed driver version. +- CDI specifications now include the NVIDIA OpenCL ICD file and the legacy `libnvidia-nvvm70.so` library when present. + OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically. +- On WSL2, CDI discovery now includes additional `.so`, `.bin`, and `.dll` files from the NVIDIA driver store instead of relying only on a fixed file list. + This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update. +- IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device. + Invalid requests fail with a clear error. +- JIT-CDI mode now honors the `nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks` configuration option. + You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible. +- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. + For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide. +- Updating `config.toml` no longer reverts previously modified options to their defaults. +- CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries. + This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. +- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script. + NRI-based deployments can invoke the hook on hosts that do not provide a shell. +- The `nvidia-cdi-refresh` systemd units are now installed and enabled using distribution-native package mechanisms, including migration of stale RPM enablement links during upgrades. + Automatic CDI specification refresh remains available more reliably after package installation and upgrades. + +### Packaging Changes + +- RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd. + The resulting packages can be installed on Amazon Linux 2. +- Source package builds can use Podman by setting `DOCKER=podman`. + The build handles Podman's local image naming, SELinux volume labeling, and artifact-directory creation automatically. + +#### Enhancements to container-toolkit Container Images + +- The `container-toolkit` image now uses the non-development distroless base and includes a static BusyBox shell. + Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image. + +### Included Packages + +The following packages are included: + +- `nvidia-container-toolkit 1.20.0` +- `nvidia-container-toolkit-base 1.20.0` +- `libnvidia-container-tools 1.20.0` +- `libnvidia-container1 1.20.0` + +The following `container-toolkit` containers are included: + +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0` +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0-packaging` + ## NVIDIA Container Toolkit 1.19.1 This release of the NVIDIA Container Toolkit `v1.19.1` is a bugfix release. diff --git a/container-toolkit/versions1.json b/container-toolkit/versions1.json index 24b46d827..b1f6d6c16 100644 --- a/container-toolkit/versions1.json +++ b/container-toolkit/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": "true", + "url": "../1.20/", + "version": "1.20" + }, + { "url": "../1.19.1/", "version": "1.19.1" }, @@ -23,49 +27,5 @@ { "url": "../1.17.8/", "version": "1.17.8" - }, - { - "url": "../1.17.7/", - "version": "1.17.7" - }, - { - "url": "../1.17.6/", - "version": "1.17.6" - }, - { - "url": "../1.17.5/", - "version": "1.17.5" - }, - { - "url": "../1.17.4/", - "version": "1.17.4" - }, - { - "url": "../1.17.3/", - "version": "1.17.3" - }, - { - "url": "../1.17.2/", - "version": "1.17.2" - }, - { - "url": "../1.17.1/", - "version": "1.17.1" - }, - { - "url": "../1.17.0/", - "version": "1.17.0" - }, - { - "url": "../1.16.2/", - "version": "1.16.2" - }, - { - "url": "../1.16.1/", - "version": "1.16.1" - }, - { - "url": "../1.16.0/", - "version": "1.16.0" } ] diff --git a/repo.toml b/repo.toml index 374e1d164..458924fd9 100644 --- a/repo.toml +++ b/repo.toml @@ -110,8 +110,8 @@ project_build_order = [ docs_root = "${root}/container-toolkit" project = "container-toolkit" name = "NVIDIA Container Toolkit" -version = "1.19.1" -source_substitutions = {version = "1.19.1"} +version = "1.20" +source_substitutions = {version = "1.20"} copyright_start = 2020 redirects = [ { path="concepts.html", target="index.html" }, From b8dedca744348c230390f3bfed2de1784b8730d0 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 5 Aug 2026 11:12:48 -0400 Subject: [PATCH 2/3] docs: Review feedback for CNT 1.20 Signed-off-by: Mike McKiernan --- container-toolkit/cdi-support.md | 27 ++------- container-toolkit/docker-specialized.md | 12 ++-- container-toolkit/index.md | 1 - container-toolkit/nri-support.md | 81 ------------------------- container-toolkit/release-notes.md | 20 +++--- 5 files changed, 24 insertions(+), 117 deletions(-) delete mode 100644 container-toolkit/nri-support.md diff --git a/container-toolkit/cdi-support.md b/container-toolkit/cdi-support.md index ccb278ae2..e548d107d 100644 --- a/container-toolkit/cdi-support.md +++ b/container-toolkit/cdi-support.md @@ -176,11 +176,11 @@ persistent specifications described in ### Disabling Hooks in JIT-CDI Mode -You can prevent JIT-CDI mode from adding specific hooks to an in-memory -specification. The supported hook names are `chmod`, `create-symlinks`, +You can prevent JIT-CDI mode from adding specific hooks to an in-memory specification. +The supported hook names are `create-symlinks`, `disable-device-node-modification`, `enable-cuda-compat`, -`update-application-profile`, and `update-ldcache`. The deprecated `chmod` hook -is disabled by default. Use `all` to disable every hook. +`update-application-profile`, and `update-ldcache`. +Use `all` to disable every hook. The following command disables the application-profile and dynamic-linker cache hooks. Separate multiple values with a colon: @@ -244,25 +244,6 @@ $ podman run --rm \ The preceding sample command requests the full GPU with index 0 and the first MIG device on GPU 1. The output should show only the UUIDs of the requested devices. -### Graphics and OpenCL Workloads - -CDI specifications include the driver libraries and configuration files that -graphics and OpenCL applications require. When the NVIDIA OpenCL installable -client driver (ICD) file is present on the host, the specification mounts it at -`/etc/OpenCL/vendors/nvidia.icd` in the container. - -CDI specifications also include the `update-application-profile` hook. At -container creation, this hook writes -`/etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf`. The profile -limits EGL and Vulkan visibility to the physical GPUs that are available in the -container. As a result, applications do not enumerate unassigned physical GPUs. - -If the application-profile hook conflicts with an application, disable it only -for that workload or environment. For JIT-CDI mode, refer to -[Disabling Hooks in JIT-CDI Mode](#disabling-hooks-in-jit-cdi-mode). For a -persistent specification, use the `--disable-hook update-application-profile` -option described in [Manual CDI Specification Generation](#manual-cdi-specification-generation). - ### IMEX Channels Containers can request specific NVIDIA IMEX channels with the diff --git a/container-toolkit/docker-specialized.md b/container-toolkit/docker-specialized.md index 465308ab9..4d134b0a9 100644 --- a/container-toolkit/docker-specialized.md +++ b/container-toolkit/docker-specialized.md @@ -218,10 +218,14 @@ $ docker run --rm --runtime=nvidia \ ``` -When CDI or JIT-CDI injects the GPU, the toolkit also limits EGL and Vulkan -visibility to the physical GPUs assigned to the container. For more information -and exceptional compatibility settings, refer to -[Graphics and OpenCL Workloads](cdi-support.md#graphics-and-opencl-workloads). +Legacy mode limits EGL and Vulkan visibility to the physical GPUs assigned to +a container when the `graphics` or `display` driver capability is enabled. +The 1.20.0 release adds the same behavior to CDI and JIT-CDI modes through the +`update-application-profile` hook. This change provides feature parity across +the modes. + +If the application-profile hook conflicts with an application, refer to +[Disabling Hooks in JIT-CDI Mode](cdi-support.md#disabling-hooks-in-jit-cdi-mode). ### Constraints diff --git a/container-toolkit/index.md b/container-toolkit/index.md index 45253d4f2..3cab94725 100644 --- a/container-toolkit/index.md +++ b/container-toolkit/index.md @@ -20,7 +20,6 @@ release-notes.md arch-overview.md Container Device Interface -Node Resource Interface docker-specialized.md ``` diff --git a/container-toolkit/nri-support.md b/container-toolkit/nri-support.md deleted file mode 100644 index d767d71b0..000000000 --- a/container-toolkit/nri-support.md +++ /dev/null @@ -1,81 +0,0 @@ -(nri-support)= - -# Support for Node Resource Interface - -## About the NRI Plugin - -The NVIDIA Container Toolkit container can run a Node Resource Interface (NRI) -plugin. The plugin connects to an NRI-enabled container runtime and adds -Container Device Interface (CDI) device requests to container configurations. -This path is useful when the runtime cannot process the original CDI request -directly. - -The plugin enforces different namespace rules for workload and management -devices: - -- The plugin can inject ordinary workload devices, such as `nvidia.com/gpu`, - into containers in any namespace. -- The plugin injects `management.nvidia.com/gpu` devices only into the toolkit - namespace or an additional namespace that an administrator explicitly - authorizes. - -## Enabling the NRI Plugin - -The container runtime must have NRI enabled, and the toolkit container must have -access to the runtime's NRI socket. Keep the toolkit installer running as a -daemon. The `--enable-nri-plugin` option is ignored when `--no-daemon` is set. - -Add the following options to the toolkit container entry point: - -```console -$ nvidia-ctk-installer \ - --enable-nri-plugin \ - --nri-namespace=nvidia-gpu-operator -``` - -The `--nri-namespace` value must match the Kubernetes namespace that contains -the toolkit pod. Use `--nri-socket` if the runtime exposes the NRI socket at a -nondefault path. - -You can configure the same settings with environment variables: - -```yaml -env: - - name: ENABLE_NRI_PLUGIN - value: "true" - - name: NRI_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace -``` - -## Authorizing Management Devices in Additional Namespaces - -Use `--nri-management-cdi-device-namespaces` to allow a namespace to receive -management CDI devices. You can repeat the option. The toolkit namespace is -always allowed and does not need to appear in the list. - -The following example authorizes management devices in two additional -namespaces: - -```console -$ nvidia-ctk-installer \ - --enable-nri-plugin \ - --nri-namespace=nvidia-gpu-operator \ - --nri-management-cdi-device-namespaces=gpu-monitoring \ - --nri-management-cdi-device-namespaces=gpu-diagnostics -``` - -For a container deployment, the equivalent environment variable accepts a -comma-separated list: - -```yaml -env: - - name: NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES - value: "gpu-monitoring,gpu-diagnostics" -``` - -Namespace matching is exact. Wildcards are not supported. Allow only namespaces -that contain trusted management workloads. A management CDI device can provide -broader access than an ordinary workload device, so follow the principle of -least privilege. diff --git a/container-toolkit/release-notes.md b/container-toolkit/release-notes.md index 394207e85..d43e4d290 100644 --- a/container-toolkit/release-notes.md +++ b/container-toolkit/release-notes.md @@ -16,32 +16,32 @@ This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. - CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container. Graphics applications no longer see unassigned host GPUs through these APIs. + For more information, refer to [issue #1899](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1899) and [PR #1939](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1939). - CUDA compatibility handling now uses `libcuda.so` ELF metadata whenever it is available. This improves CUDA minor-version compatibility by selecting the container's compatibility libraries only when they are appropriate for the installed driver. - Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version. This fix avoids both missing required libraries and injecting libraries from another installed driver version. + For information about multiple-directory discovery, refer to [issue #1559](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1559) and [PR #1820](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1820). For information about exact-version matching, refer to [PR #1948](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1948). - CDI specifications now include the NVIDIA OpenCL ICD file and the legacy `libnvidia-nvvm70.so` library when present. OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically. + For information about OpenCL support, refer to [issue #682](https://github.com/NVIDIA/nvidia-container-toolkit/issues/682) and [PR #1893](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1893). For information about legacy NVVM support, refer to [issue #1875](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1875) and [PR #1876](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1876). - On WSL2, CDI discovery now includes additional `.so`, `.bin`, and `.dll` files from the NVIDIA driver store instead of relying only on a fixed file list. This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update. + For more information, refer to [issue #1864](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1864) and [PR #1890](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1890). - IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device. Invalid requests fail with a clear error. + For more information, refer to [issue #1309](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1309) and [PR #1913](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1913). - JIT-CDI mode now honors the `nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks` configuration option. You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible. -- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. - For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide. - Updating `config.toml` no longer reverts previously modified options to their defaults. - CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries. This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. -- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script. - NRI-based deployments can invoke the hook on hosts that do not provide a shell. -- The `nvidia-cdi-refresh` systemd units are now installed and enabled using distribution-native package mechanisms, including migration of stale RPM enablement links during upgrades. - Automatic CDI specification refresh remains available more reliably after package installation and upgrades. + For more information, refer to [issue #373](https://github.com/NVIDIA/nvidia-container-toolkit/issues/373) and [PR #1894](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1894). ### Packaging Changes - RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd. - The resulting packages can be installed on Amazon Linux 2. + The resulting packages can be installed on older Linux distributions that do not support zstd compression, such as Amazon Linux 2. - Source package builds can use Podman by setting `DOCKER=podman`. The build handles Podman's local image naming, SELinux volume labeling, and artifact-directory creation automatically. @@ -49,6 +49,10 @@ This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. - The `container-toolkit` image now uses the non-development distroless base and includes a static BusyBox shell. Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image. +- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. + For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide. +- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script. + NRI-based deployments can invoke the hook on hosts that do not provide a shell. ### Included Packages @@ -105,7 +109,7 @@ v0.7.0 of the CDI spec schema. The container runtime support for v0.7.0 of the C - podman >= 5.1.0 - [containers/podman@a40cf31](https://github.com/containers/podman/commit/a40cf3195acb6ac5fea5ab4617afb99006a3bed7) - crio >= 1.30.0 - [cri-o/cri-o@fd9aa76](https://github.com/cri-o/cri-o/commit/fd9aa76250fe05625d8c968b922cd1a0ae88eb1b) -If you are using a container runtime version that does not support v0.7.0 of the CDI spec schema, +If you are using a container runtime version that does not support v0.7.0 of the CDI spec schema, it is recommended to set the `no-additional-gids-for-device-nodes` CDI feature flag so that an older version of the CDI spec schema is used for spec file generation: From 3b2653adc16d55dd5d8f5215d6a5115452602392 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 5 Aug 2026 12:03:47 -0400 Subject: [PATCH 3/3] docs: Add RN for setting default runtime_type Signed-off-by: Mike McKiernan --- container-toolkit/release-notes.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/container-toolkit/release-notes.md b/container-toolkit/release-notes.md index d43e4d290..cd70b5046 100644 --- a/container-toolkit/release-notes.md +++ b/container-toolkit/release-notes.md @@ -37,6 +37,15 @@ This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. - CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries. This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. For more information, refer to [issue #373](https://github.com/NVIDIA/nvidia-container-toolkit/issues/373) and [PR #1894](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1894). +- Fixed an issue where NVIDIA runtime handlers in a generated containerd + drop-in configuration could omit `runtime_type` when the base configuration + did not define it. + The affected containers failed to start with a + `container.Runtime.Name must be set` error. + The toolkit now sets the default runtime type when the field is missing or empty. + For more information, refer to + [issue #1956](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1956) + and [PR #1969](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1969). ### Packaging Changes