Benchmark Multi-GPU setups

[geerlingguy]

I'd like to see how multi-GPU setups perform both on a Pi using the NVIDIA/open-gpu-kernel-modules#972 patch — and then compare those numbers to my Intel Core Ultra 265K system, to see the difference.

See: geerlingguy/raspberry-pi-pcie-devices#504

[geerlingguy]

Pi CM5 Dual GPU setup - RTX 4070 Ti + RTX A4000 (Pi PCIe Default Config)

Total VRAM is 12GB + 16GB (28GB)

Idle power draw: 37W

Llama-3.2-3B-Instruct-Q4_K_M.gguf (1.87GB)

build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -fa 1 -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2 -mmp 0

model	size	params	backend	ngl	fa	mmap	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp512	2580.68 ± 1525.63
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096	4003.43 ± 0.72
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	tg128	104.16 ± 5.54
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096+tg128	1969.54 ± 10.44

Power consumption: 233W

gpt-oss-20b-Q4_K_M.gguf (10.81GB)

model	size	params	backend	ngl	fa	mmap	test	t/s
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp512	1845.27 ± 715.03
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp4096	3168.88 ± 9.09
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	tg128	74.64 ± 10.02
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp4096+tg128	1634.00 ± 0.32

Power consumption: 286.1W

Qwen_Qwen3-30B-A3B-Q4_K_M.gguf (18.40GB)

model	size	params	backend	ngl	fa	mmap	test	t/s
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp512	635.61 ± 133.88
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096	881.76 ± 1.28
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	tg128	19.43 ± 0.13
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096+tg128	361.56 ± 1.48

Power consumption: 342.2W

Moral of the story so far is you can load larger models than will fit on a single GPU with limited VRAM, but they will not run faster with all the Pi defaults. Going to try to tinker with some PCIe settings next.

[geerlingguy]

Pi CM5 Dual GPU setup - RTX 4070 Ti + RTX A4000 ('ACS Disabled' PCIe Config)

Total VRAM is 12GB + 16GB (28GB)

Idle power draw: 38W

See: geerlingguy/raspberry-pi-pcie-devices#791

Qwen_Qwen3-30B-A3B-Q4_K_M.gguf (18.40GB)

model	size	params	backend	ngl	fa	mmap	test	t/s
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp512	727.83 ± 2.14
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096	880.95 ± 2.01
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	tg128	19.43 ± 0.17
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096+tg128	361.42 ± 1.48

Power consumption: 296.2W

After my testing in that issue linked above, it seems like the switch preconfigured from Dolphin already has the device-to-device access working, though I'm wondering if llama.cpp isn't taking full advantage there. I may try vLLM as well.

[geerlingguy]

I tried installing vllm, but got a fail:

uv venv vllm-env
source vllm-env/bin/activate
uv pip install -U vllm --torch-backend=auto
...

      [stderr]
      2025-12-10 15:08:38,145 - scikit_build_core - INFO - RUN: /usr/bin/cmake -E capabilities
      2025-12-10 15:08:38,161 - scikit_build_core - INFO - CMake version: 3.31.6
      2025-12-10 15:08:38,161 - scikit_build_core - INFO - Implementation: cpython linux on aarch64
      2025-12-10 15:08:38,165 - scikit_build_core - INFO - Build directory:
      /home/jgeerling/.cache/uv/sdists-v9/pypi/xgrammar/0.1.27/in8fW3rFwEQrHuacSIHMC/src/build
      2025-12-10 15:08:38,185 - scikit_build_core - INFO - RUN: /usr/bin/ninja --version
      2025-12-10 15:08:38,187 - scikit_build_core - INFO - Ninja version: 1.12.1
      2025-12-10 15:08:38,189 - scikit_build_core - INFO - RUN: /usr/bin/cmake
      -S. -Bbuild -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -Cbuild/CMakeInit.txt
      -DCMAKE_INSTALL_PREFIX=/tmp/tmpxka2tyn8/wheel/platlib/xgrammar -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja
      2025-12-10 15:08:39,097 - scikit_build_core - INFO - RUN: /usr/bin/cmake --build build -v

      *** CMake build failed

      hint: This usually indicates a problem with the package or the build environment.
  help: `xgrammar` (v0.1.27) was included because `vllm` (v0.12.0) depends on `xgrammar`

[geerlingguy]

Pi CM5 Dual GPU setup - RX 7900 XT + Radeon AI Pro R9700

Total VRAM is 20GB + 32GB (52GB)

Idle power draw: 30W

Llama-3.2-3B-Instruct-Q4_K_M.gguf (1.87GB)

build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -fa 1 -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2 -mmp 0

model	size	params	backend	ngl	fa	mmap	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp512	737.19 ± 1.89
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096	707.43 ± 1.61
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	tg128	47.62 ± 0.22
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096+tg128	109.90 ± 0.22

Power consumption: 296.6W

gpt-oss-20b-Q4_K_M.gguf (10.81GB)

model	size	params	backend	ngl	mmap	test	t/s
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp512	30.13 ± 0.25
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp4096	328.81 ± 1.31
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	tg128	14.74 ± 0.00
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	0	pp4096+tg128	85.94 ± 0.21

Power consumption: 320W

Qwen_Qwen3-30B-A3B-Q4_K_M.gguf (18.40GB)

TODO...

radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
[New LWP 2349]
[New LWP 2318]
[New LWP 2317]
[New LWP 2316]
[New LWP 2315]
[New LWP 2314]

Power consumption: TODOW

Llama-3.3-70B-Instruct-Q4_K_M.gguf (40GB)

TODO...

radv/amdgpu: The CS has been cancelled because the context is lost. This context is guilty of a hard recovery.
[New LWP 2488]
[New LWP 2487]
[New LWP 2486]
[New LWP 2485]
[New LWP 2484]
[Thread debugging using libthread_db enabled]

Power consumption: TODOW

Those errors (CS has been cancelled because the context is lost) seem to happen after a few minutes of model loading time... so I wonder if a newer kernel version might help. I'm going to re-flash Pi OS and install the latest version of the AMD PR:

jgeerling@cm5:~/Downloads/llama.cpp $ uname -a
Linux cm5 6.17.10-v8-16k+ #1 SMP PREEMPT Thu Dec  4 18:31:39 UTC 2025 aarch64 GNU/Linux
jgeerling@cm5:~/Downloads/llama.cpp $ sudo rpi-update pulls/7113
jgeerling@cm5:~/Downloads/llama.cpp $ sudo reboot
...
jgeerling@cm5:~/Downloads/llama.cpp $ uname -a
Linux cm5 6.18.0-v8-16k+ #1 SMP PREEMPT Tue Dec  9 14:44:34 UTC 2025 aarch64 GNU/Linux

But now the AI Pro R9700 isn't initializing... seems like a regression in 6.18.y, maybe?

[geerlingguy]

Intel Core Ultra 265K Dual GPU setup - RTX 4070 Ti + RTX A4000

Total VRAM is 12GB + 16GB (28GB)

Idle power draw: 56.5W

Llama-3.2-3B-Instruct-Q4_K_M.gguf (1.87GB)

build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -fa 1 -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2 -mmp 0

model	size	params	backend	ngl	fa	mmap	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp512	831.51 ± 267.15
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096	888.43 ± 19.14
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	tg128	131.47 ± 18.62
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	Vulkan	99	1	0	pp4096+tg128	742.05 ± 38.51

Power consumption: 160.2W

gpt-oss-20b-Q4_K_M.gguf (10.81GB)

model	size	params	backend	ngl	fa	mmap	test	t/s
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp512	1039.47 ± 413.51
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp4096	1292.87 ± 38.47
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	tg128	127.34 ± 1.00
gpt-oss 20B Q4_K - Medium	10.81 GiB	20.91 B	Vulkan	99	1	0	pp4096+tg128	989.23 ± 14.59

Power consumption: 212.9W

Qwen_Qwen3-30B-A3B-Q4_K_M.gguf (18.40GB)

model	size	params	backend	ngl	fa	mmap	test	t/s
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp512	706.08 ± 1.17
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096	798.85 ± 1.02
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	tg128	19.49 ± 0.19
qwen3 32B Q4_K - Medium	18.40 GiB	32.76 B	Vulkan	99	1	0	pp4096+tg128	350.10 ± 0.44

Power consumption: 321.5W

Moral of the story so far is you can load larger models than will fit on a single GPU with limited VRAM, but they will not run faster with all the Pi defaults. Going to try to tinker with some PCIe settings next.