Performance of llama.cpp with Vulkan #10879
Replies: 285 comments 460 replies
-
|
AMD FirePro W8100
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
With FA:
|
Beta Was this translation helpful? Give feedback.
-
|
AMD RX 470
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
With FA:
|
Beta Was this translation helpful? Give feedback.
-
|
i got the mining edition 8 gb ran slightly better
build: e288693 (6242) |
Beta Was this translation helpful? Give feedback.
-
|
Here's a rerun with the recent flash attention updates:
|
Beta Was this translation helpful? Give feedback.
-
|
Nice, it seems to help even on Polaris. |
Beta Was this translation helpful? Give feedback.
-
|
ubuntu 24.04, vulkan and cuda installed from official APT packages.
build: 4da69d1 (4351) vs CUDA on the same build/setup
build: 4da69d1 (4351) |
Beta Was this translation helpful? Give feedback.
-
|
Macbook Air M2 on Asahi Linux ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
For the record I think this is slow on the HoneyKrisp side rather than llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Can you share how you got vulkan to build on Asahi? I can't seem to get cmake to notice it. |
Beta Was this translation helpful? Give feedback.
-
|
Spoke too soon, got it working! |
Beta Was this translation helpful? Give feedback.
-
|
Gentoo Linux on ROG Ally (2023) Ryzen Z1 Extreme ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: Found 4 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
Cool setup! Could you also post the result of 1, 2 and 3 7900 XTX GPUs? You can use only the first GPU with |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
For this multi GPU case getting Vulkan to support #6017 pipeline parallelism might help improve the prompt processing speed. |
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve I updated the commit id in all my result. |
Beta Was this translation helpful? Give feedback.
-
|
build: 0d52a69 (4439) NVIDIA GeForce RTX 3090 (NVIDIA)
AMD Radeon RX 6800 XT (RADV NAVI21) (radv)
AMD Radeon (TM) Pro VII (RADV VEGA20) (radv)
Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver)
|
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve Some of the tg results here are a little low, I think they might be debug builds. The cmake step (at least on Linux) might require |
Beta Was this translation helpful? Give feedback.
-
|
I've added My release numbers for the RX 470 are basically identical to the ones I posted earlier without the flag.
|
Beta Was this translation helpful? Give feedback.
-
|
Maybe not in your case, but some other results are suspiciously low in tg (for example the RTX 3080) |
Beta Was this translation helpful? Give feedback.
-
|
Build: 8d59d91 (4450)
Lack of proper Xe coopmat support in the ANV driver is a setback honestly.
edit: retested both with the default batch size. |
Beta Was this translation helpful? Give feedback.
-
I put my A770 into a Windows PC and gave Intel GPA and vtune a shot: GPA just crashes most of the time, I couldn't get it to trace anything useful. vtune works, but does not support Vulkan. It just shows some high-level metrics in that case, not really useful sadly. |
Beta Was this translation helpful? Give feedback.
-
I did build it with cmake with build type Release. |
Beta Was this translation helpful? Give feedback.
-
|
In that case it's something else, cause it should be performing similarly to my A770. I suspect the mesa version, there was something in newer mesa versions that slowed down tg on Intel. |
Beta Was this translation helpful? Give feedback.
-
|
A750 has 448 CUs, A770 has 512 CUs I think. Personally, I am not worried about tg. I am worried about pp here. The gemm batch quickly saturates my GPU. |
Beta Was this translation helpful? Give feedback.
-
|
@0cc4m https://gitlab.freedesktop.org/mesa/mesa/-/issues/12585 |
Beta Was this translation helpful? Give feedback.
-
|
Here's something exotic: An AMD FirePro S10000 dual GPU from 2012 with 2x 3GB GDDR5. build: 914a82d (4452)
|
Beta Was this translation helpful? Give feedback.
-
|
Very interesting, and looks like it's pretty close to the W8100 in tg despite being a dual GPU card. Your backend scales pretty well with layer splitting which is why I find it worthwhile to run my RX470 and W8100 together (I end up getting results that are close to the average of both cards).
|
Beta Was this translation helpful? Give feedback.
-
|
Latest arch with For the sake of consistency I run every bit in a script and also build every target from scratch (for some reason kill -STOP -1
timeout 240s $COMMAND
kill -CONT -1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none
build: ff3fcab (4459)
This bit seems to underutilise both GPU and CPU in real conditions based on
|
Beta Was this translation helpful? Give feedback.
-
|
Unless you reduce the number of GPU layers, threads and openblas/non-openblas is not gonna make any difference. Try it with ngl 0, then only prompt processing is accelerated using Vulkan, the rest runs on CPU. This is often a good setting for integrated GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
That's something I didn't think about, with
build: ba8a1f9 (4460)
|
Beta Was this translation helpful? Give feedback.
-
|
It seems latest patches has improved the results a bit: ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
A few months later and I get:
build: f3a4b16 (5568) I run it on Linux (Arch with lama.cpp-vulkan-git package compiled by GCC 15). From my tests, only Vulkan backend ( I'm curious why I cannot go over 6 t/s. Is this an issue with the newer llama.cpp version or with my OS configuration? |
Beta Was this translation helpful? Give feedback.
-
|
Intel ARC A770 on Windows:
build: ba8a1f9 (4460) |
Beta Was this translation helpful? Give feedback.
-
Single GPU VulkanRadeon Instinct MI25 ggml_vulkan: 0 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Radeon PRO VII ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Multi GPU Vulkanggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Single GPU RocmDevice 0: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
build: 2739a71 (4461) Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Multi GPU RocmDevice 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Layer split
build: 2739a71 (4461) Row split
build: 2739a71 (4461) Single GPU speed is decent, but multi GPU trails Rocm by a wide margin, especially with large models due to the lack of row split. |
Beta Was this translation helpful? Give feedback.
-
|
What is the power profile for this MI25? Mine is 110W but its running slower than yours on git from today. |
Beta Was this translation helpful? Give feedback.
-
|
Mine defaults to 220w. |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 5700 XT on Arch using mesa-git and setting a higher GPU power limit compared to the stock card.
I also think it could be interesting adding the flash attention results to the scoreboard (even if the support for it still isn't as mature as CUDA's).
|
Beta Was this translation helpful? Give feedback.
-
|
There is no Vulkan flash attention support (except with coopmat2 on very new nvidia drivers). What you're measuring here is a CPU fallback. |
Beta Was this translation helpful? Give feedback.
-
|
I see, I was sure about the CPU fallback but didn't know there was no flash attention support at all. |
Beta Was this translation helpful? Give feedback.
-
|
I tried but there's nothing after 1 hrs , ok, might be 40 mins... Anyway I run the llama_cli for a sample eval...
Meanwhile OpenBLAS |
Beta Was this translation helpful? Give feedback.
-
|
Even at below 1t/s |
Beta Was this translation helpful? Give feedback.
-
|
Truth is ... (0.79 tokens per second), 3788.63 ms / 3 tokens So it's not even...it just slower... |
Beta Was this translation helpful? Give feedback.
-
|
❯ ~/llama.cpp/build-vulkan/bin/llama-bench -m ~/llama-2-7b.Q4_0.gguf -ngl 99 -fa 1 -dev Vulkan0
build: 0d0764d (8892) |
Beta Was this translation helpful? Give feedback.
-
|
Framework 13 running AMD Ryzen 5 7640U w/ Radeon 760M Graphics ./build/bin/llama-bench -m ~/Downloads/llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1
build: 187a456 (8910) |
Beta Was this translation helpful? Give feedback.
-
|
"int dot = 0" and "KHR_coopmat"? Weird... Maybe your SDK is not updated? But the performance looked fine? |
Beta Was this translation helpful? Give feedback.
-
|
@pt13762104 - I've been toying quite a bit recently with lots of random installs/llama.cpp flags etc to see what can be done locally even if it's slow and hit weird cases (e.g. rocm kernel drivers are hard-locked at v6 while the userspace is v7 when using the amdgpu-install script) so I'm not really sure how nicely (assume you mean vulkan SDK) will play on this machine If you've any requests for more info - happy to provide them: |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I tried a newer build and set Windows 10 to "Best performance", but same result: GPU power draw goes over 100 W and clock up to 1127 MHz while the test runs, no PerfCap while pp. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah that's odd, I don't know what the problem is. Is it like this with a smaller model as well? |
Beta Was this translation helpful? Give feedback.
-
|
Seems so (8 GB VRAM): |
Beta Was this translation helpful? Give feedback.
-
|
Check you pcie connection then? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
❯ ~/llama.cpp/build-vulkan/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 1e9d771 (8768) |
Beta Was this translation helpful? Give feedback.
-
|
Yep, I'll play...big uplift in PP compared with the R9700 in the leaderboard, small increase in TG. |
Beta Was this translation helpful? Give feedback.
-
|
❯ ~/llama.cpp/build-vulkan/bin/llama-bench -m ~/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -dev Vulkan0
build: d05fe1d (9010) |
Beta Was this translation helpful? Give feedback.
-
|
where are 8060 results? You only provided 3090 |
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: 1 = Tesla P40 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 0 = AMD Radeon RX 5700 XT (RADV NAVI10) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
build: 7b8443a (8966) |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it does! WITH the p40 at the same time. |
Beta Was this translation helpful? Give feedback.
-
|
Okay I have no idea what you mean there, as if the dev is Vulkan0 it means that only Vulkan0 is being used. If there's no device set or if you have dev as Vulkan0/Vulkan1 then you're using both cards with split mode. |
Beta Was this translation helpful? Give feedback.
-
|
I mean that when I use an actual session, both GPUs combine under Vulkan. |
Beta Was this translation helpful? Give feedback.
-
|
Oh nothing's broken. If you have multiple GPUs the model will automatically run on all of them using layer splitting unless you change the settings. Like in my case: default settings, runs on both cards with layer splitting
|
Beta Was this translation helpful? Give feedback.
-
|
That explains it then. :-) Thanks! |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Interestingly I get wildly different scores between Windows and Linux on an Arc Pro B50 ggml_vulkan: 0 = Intel(R) Arc(TM) Pro B50 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat on Linux (Ubuntu 25.10) intel i7-8700
in Windows 11 - Ryzen 5700X
build: aa46bda (9437)
|
Beta Was this translation helpful? Give feedback.
-
|
That's a known issue, because of different drivers. Intel puts a lot more work into the Windows driver optimization, sadly. |
Beta Was this translation helpful? Give feedback.
-
|
Intel Arc A770 🤷 Ubuntu 25.10 (7.0.6-070006-generic)
build: 22cadc1 (9439) |
Beta Was this translation helpful? Give feedback.
-
Vega 64 (undervolted)Sapphire RX Vega 64 (reference blower-fan type) Result$ RADV_PERFTEST=nogttspill llama-bench --model ./llama-2-7b.Q4_0.gguf -ngl 100 -fa 1 -r 5 --delay 5
build: 5dcb711 (9464) LACT profile:Details[...]
profiles:
Undervolt lowest MAX perf:
gpus:
1002:687F-1002:6B76-0000:2f:00.0:
fan_control_enabled: true
fan_control_settings:
mode: static
static_speed: 1.0
temperature_key: edge
interval_ms: 500
curve:
54: 0.0
55: 0.05
60: 0.2
70: 0.7
80: 0.9
90: 1.0
spindown_delay_ms: 10000
change_threshold: 3
power_cap: 330.0
performance_level: manual
gpu_vf_curve:
5:
voltage: 825
clockspeed: 1400
2:
voltage: 803
6:
voltage: 850
clockspeed: 1520
7:
voltage: 1050
clockspeed: 1580
4:
voltage: 825
1:
voltage: 802
0:
voltage: 801
3:
voltage: 825
mem_vf_curve:
3:
voltage: 980
clockspeed: 1050
power_profile_mode_index: 5
power_states:
core_clock:
- 7
memory_clock:
- 3Vega 64 (default profile)ggml_vulkan: Found 1 Vulkan devices:
build: 5dcb711 (9464) |
Beta Was this translation helpful? Give feedback.
-
|
$> uname -a $> llama-bench -fa 0,1 -ngl 666 -m /zfast/llm/coding/llama-2-7b.Q4_0.gguf ggml_vulkan: Found 2 Vulkan devices:
build: a731805 (9493) Removing BLAS (OpenBLAS) does not decrease results. llama-bench -fa 0,1 -ngl 666 -m /zfast/llm/coding/llama-2-7b.Q4_0.gguf
build: ad1b88c (9525) Gentoo: ggml_vulkan: Found 2 Vulkan devices:
build: 65ef50a (9501) |
Beta Was this translation helpful? Give feedback.
-
|
Your link points to this thread... |
Beta Was this translation helpful? Give feedback.
-
|
My mistake, sorry. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Hey in the future please don't put standalone CPU results as a top level reply as it might confuse someone. I've hidden it for now. |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
./llama-bench -fa 0,1 -m ../../models/llama-2-7b.Q4_0.gguf
build: 8ed274e (9630) also OpenVINO backend OpenVINO: using device GPU
build: 8ed274e (9630) SYCL bench results |
Beta Was this translation helpful? Give feedback.
-
|
that's about as fast as a 2060, which isn't slow for a mobile GPU like this... I wonder what can the new PTL GPU do. Suprisingly Vulkan and SYCL roughly performed the same and is about 2.2x slower than OpenVINO... |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 580 2048SP (mining edition, 8GB) — Windows, AMD proprietary driver ggml_vulkan: 0 = AMD Radeon RX 580 2048SP (AMD proprietary driver) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | int dot: 0 | matrix cores: none
Noticeably lower than your RX 470 mining edition above (same 2048SP silicon) — almost certainly the Windows proprietary driver vs RADV on Linux, I've seen close to 2x gaps between the two on this exact card on other workloads. Will follow up with a Linux/RADV run on the same hardware for a direct same-chip comparison. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah the Windows driver for Polaris is too old and unsupported. There are also a couple bugs with it and llama.cpp which haven't been fixed, for those old cards you should be using Linux and Mesa. |
Beta Was this translation helpful? Give feedback.
-
|
That tracks with what we've seen — appreciate the confirmation. Will get a Linux/Mesa RADV run on this same card up soon for a direct comparison, then. |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Apple Silicon benchmark thread, but for Vulkan! We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our Vulkan releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM. If you use RADV please run with the environment variableRADV_PERFTEST=nogttspillas that can fix a bunch of performance issues.Share your llama-bench results along with the git hash and Vulkan info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same setup I'll prioritize newer commits with substantial Vulkan updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that the memory speed and number of channels will greatly affect your inference speed!
Vulkan Scoreboard (Click on the headings to expand the section)
Llama 2 7B, Q4_0, no FA
Llama 2 7B, Q4_0, FA enabled
Beta Was this translation helpful? Give feedback.
All reactions