Allow the device plugin to register multiple identical devices sharing the same IOMMU group

[andreyod]

As it is today, the device plugin only advertises one device per IOMMU group.
Some hardware boards by default can have more than 1 GPU or NVSwitch per IOMMU group.
In this case the number of allocatable devices on the node will be equal to the number of IOMMU groups and will be less than the actual number of devices.
We have a real customer use case to create a single Kubevirt VM on a node and passthrough all Nvidia GPUs and NVSwitches, but currently this is not possible.

The proposal is to allow the device plugin to conditionally register all the devices when explicitly asked by the deployer(for example by setting a CLI flag).

For more details and implementation reference, see: kubevirt/kubevirt#13347

[andreyod]

Please let me know if this is acceptable and we will implement and open a PR for this feature.

[ezrasilvera]

Just to add to that - we also implemented and tested such a change on this code and it indeed fixed the issue and the changes seems relatively simple.
As @andreyod mention we understand the reasoning behind the design decision of trying to block the user from using devices from the same IOMMU group in two different VMs but this also blocks legitimate use cases that work perfectly when using qemu/kvm directly (not through KubeVirt).
We can leave the default behavior as is and enable the "full discovery" only if explicitly requested by the user.

[andreyod]

Hi @rthallisey . WDYT?

[zvonkok]

Independent of the shortcomings of the device-plugin how it handles DGX, HGX systems your IOMMU groups look off. All of those components should be each in its own IOMMU Group.
Having all those components in one IOMMU Group means only one VM can allocate them. Have you enabled ACS? What are you running as the host OS and kernel?

Depending on the number of GPUs you pass-through into a single VM it will depend how many NVSwitch'es you also need to pass-through.

What is the NVSwitch virtualization model you're using? Looking through the output from the PR, you have A100 is it a DGX or HGX?

Have you reviewed the documentation at https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html?

This is related to the DRA topic @alaypatel07 @rmohr @rthallisey

[ezrasilvera]

@zvonkok Thanks for the input - Few comments:

1. We did enable ACS to test. Indeed for the A100 board it separated the GPUs and NVSwitches, however for the H100 it separated only the GPUs and left all NVSwitches in one IOMMU group which means KubeVirt will show us only one NVSWitch instead of 4
2. To support multiple VMs we are using shared NVSwitches (using NVidia partitions) so basically all NVSwitches are passedthrough into the service VM and not to the user VMs. This means that we have no problem in having 4 NVSwitches in one IOMMU group as we assign them into a single (service) VM
3. While its true ACS may solve the issue for some boards if we enable ACS we won't be able to use GPUDirect RDMA which will hurt our node-to-node performance as well as GPU to storage performance

I want to emphasis that we are now looking to support a very specific use case where we use single VM per node and assign all GPUs and NVSwitches into that VM. This is fully supported if we use KVM/QEMU directly without the need to enable ACS (as we use single VM) .
Is there any special reason why not enable an optional feature/flag that will allow the device plugin to discover all devices?

BTW, please note that the current design doesn't really protect users as it's working according to the device type - which means if we have NVSwitch and GPU in the same group a user can still try to assign GPU to one VM and NVSWitch to anothe VM and get a failure .

[zvonkok]

I understand that you have only one use case and pass all GPUs and NVSwitches through into the VM. The question here is how the device plugin should handle that.

Usually, you would have something like this:

IOMMU Group 16:
	61:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 25:
	52:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 34:
	43:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 67:
	1b:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 89:
	df:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 98:
	d1:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 107:
	c3:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 140:
	9d:00.0 3D controller [0302]: NVIDIA Corporation GH100 [H100 SXM5 80GB] [10de:2330] (rev a1)
--
IOMMU Group 164:
	07:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
IOMMU Group 165:
	08:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
IOMMU Group 166:
	09:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
IOMMU Group 167:
	0a:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)

Are we saying that for every DGX/HGX system, we always want to pass through all the GPUs and NVSwitches?

[ezrasilvera]

We are mainly saying that because we don't know what the user wants to do we shouldn't filter devices to "protect the user" or at least allow a mode that discovers snd exposes all devices if the user request it explicitly.

BTW, what about the GPUDirect RDMA issue with ACS?
Also as I pointed out in the shared MVSwitches mode that NVIdia recommends we assign only the nvswitches to the special service VM so we basically don't need to enable ACS and still partition the nodd into multiple VMs

[ezrasilvera]

Hi @zvonkok . I enabled the ACS on our SupeMicro but unfortunetly the NVSwitches are still in one group (the GPUs are in separate groups)

IOMMU Group 130 00:10.0 PCI bridge [0604]: Intel Corporation Device [8086:1bb0] (rev 11)
Kernel driver in use: pcieport
IOMMU Group 130 03:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 03:00.1 Memory controller [0580]: PMC-Sierra Inc. Device [11f8:4128]
Subsystem: NVIDIA Corporation Device [10de:1643]
IOMMU Group 130 04:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:01.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:02.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:03.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 05:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 06:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 07:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 08:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci

[zvonkok]

BTW, what about the GPUDirect RDMA issue with ACS?

There is no issue with ACS and GPUDirect RDMA, you need to configure ACS, ATS properly depending on who talks to whom.

[zvonkok]

We have several overlapping projects that require this functionality, and there is no consensus on whether we should invest effort in extending the device plugin, which is currently insufficient to cover all the use cases that we might encounter on DGX/HGX-like systems.

Another idea that is being considered is to invest effort in a capable DRA driver/plugin, which is the path forward for all GPU-related allocations in a Kubernetes cluster.

A straightforward and effective way to expose those devices is to ignore all IOMMU groups and expose each vendor:10de + device-id (GPU + NvSwitch) as an extended resource. This will not give us any logical grouping, just all the VFIO devices as a single extended resource.

[ezrasilvera]

Thanks @zvonkok . We have looked into DRA and indeed it's a good direction we are investigation as well.
However we need something relatively urgent for some customer use-case and a simple change to the device-plugin (we can derive the PR) can help us (and also DRA is still Beta in k8s )
In general I think the current design of exposing limited resources (one per IOMMU group) is not a good approach especially that this scenario on this board configuration can be easily achieved with libvirt/KVM (without KubeVirt) .
Do you see any special reason why not to add an optional flag that allow discovery of all devices ?

[zvonkok]

In general I think the current design of exposing limited resources (one per IOMMU group) is not a good approach especially that this scenario on this board configuration can be easily achieved with libvirt/KVM (without KubeVirt) .

@ezrasilvera As mentioned before, your system does not properly set up ACS, and there are different reasons. Having said that, I am in favour of having the capability in the device plugin to use on HGX/DGX-like systems.

Please do not rely on your IOMMU Group setup, since this will break other systems.
What combination of GPUs and NVSwitches do we want to support?

If we want all GPUs and all NVSwitches, rather than parsing or relying on the IOMMU Group, expose every GPU and NvSwitches that are bound to vfio-pci.

Can you please post a PR for review? I will tag the necessary people to move this forward. I will also need to test this on my side to ensure it doesn't break other use cases.

[andreyod]

@ezrasilvera @zvonkok
PR #141 adds support to register all devices.