Results not good for my GPU

[simin75simin]

Description

I have 3060 and 32GB of RAM, but it gives me very old models, as shown below. I think quantized qwen3.6 and Gemma 4 are much better

╭─────────────────────────────────────────────────── Hardware Info ────────────────────────────────────────────────────╮
│ GPU 0: NVIDIA GeForce RTX 3060 — 12.0 GB (CC 8.6, CUDA 12.6) — BW: 360 GB/s │
│ CPU: Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz — 6 cores (AVX2) │
│ RAM: 31.9 GB │
│ Disk free: 100.9 GB │
│ OS: windows │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

                                           Recommended Models

┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ # ┃ Model ┃ Params ┃ Quant ┃ Published ┃ Downloads ┃ Score ┃ License ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 1 │ openai/gpt-oss-20b │ 21.5B │ Q3_K_M │ 2025-08-04 │ 6.8M │ 74.4 │ apache-… │
│ │ │ (3.6B… │ │ │ │ │ │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 2 │ Qwen/Qwen3-14B │ 14.8B │ Q5_K_M │ 2025-04-27 │ 1.6M │ 74.0 │ apache-… │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 3 │ microsoft/phi-4 │ 14.7B │ Q5_K_M │ 2024-12-11 │ 810.0K │ 71.6 │ mit │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 4 │ Qwen/Qwen3-8B │ 8.2B │ Q5_K_M │ 2025-04-27 │ 11.0M │ 66.2 │ apache-… │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 5 │ deepseek-ai/DeepSeek-R1-Distill-Qwen-14B │ 14.8B │ Q5_K_M │ 2025-01-20 │ 499.5K │ 61.2 │ mit │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 6 │ google/gemma-3-12b-it │ 12.2B │ Q5_K_M │ 2025-03-01 │ 2.8M │ 60.8 │ gemma │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 7 │ deepseek-ai/DeepSeek-R1-Distill-Llama-8B │ 8.0B │ Q5_K_M │ 2025-01-20 │ 443.8K │ 56.1 │ mit │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 8 │ Qwen/Qwen2.5-14B-Instruct │ 14.8B │ Q5_K_M │ 2024-09-16 │ 2.0M │ 56.0 │ apache-… │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 9 │ Qwen/Qwen3.6-27B │ 27.8B │ Q4_K_M │ 2026-04-21 │ 5.3M │ 66.4 │ apache-… │
├─────┼──────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│ 10 │ google/gemma-2-9b-it │ 9.2B │ Q5_K_M │ 2024-06-24 │ 394.8K │ 50.6 │ gemma │
└─────┴──────────────────────────────────────────┴────────┴────────┴────────────┴───────────┴───────┴──────────┘
Top pick confidence: Low (direct benchmark but very close (+0.4))
Benchmark reference: 2026-05 curated snapshot; live AA / LiveBench / Aider merged when reachable.
Note: Top candidates are very close (#1 vs #2: 0.4 pts).

Steps to Reproduce

whichllm --gpu "RTX 3060"

Hardware Info

Python Version

No response

Operating System

No response

whichllm Version

No response

[samarthpatel24]

The results look alright for a 3060 12GB. Qwen3.6-27B is in your list at # 9, it just gets pushed down because at Q4_K_M the ranker penalizes the quantization.

For Gemma 4, try running with --refresh — it does show up but with a ~ next to the score, meaning no curated benchmark data yet so the ranker discounts it heavily.

[Andyyyy64]

You're right, and thanks for the detailed report. This is a ranking problem on our end, not your setup.

Qwen3.6-27B has the highest benchmark score of anything in your list (83.5, against 66.7 for the next one). It lands at #9 only because it does not fully fit in 12GB, and the current penalty for partial offload is too aggressive, so it drops below smaller models that fit fully. On a 3060 12GB with 32GB RAM it would run at partial offload, slower but usable, and it should sit near the top with that caveat rather than buried.

@samarthpatel24 thanks for jumping in here. The quant step is part of the picture; the bigger factor in this case is the VRAM fit rather than Q4_K_M itself, since the Q4 penalty is small. And you're right about Gemma 4: it has no curated benchmark yet, so the ranker discounts it, which is why it shows with the ~ marker.

Tracking the fix in a separate issue now, and I'll get the partial-offload calibration sorted out.

[Andyyyy64]

@simin75simin I opened #108 for this.

The fix removes the extra full-GPU sort bonus that was pushing Qwen3.6-27B down after it had already been penalized for partial offload. With the local fix, the 3060 12GB case puts Qwen3.6-27B back in the top group with a clear partial-offload warning instead of burying it under weaker full-GPU models.

I also checked the opposite cases so this does not swing too far the other way: A3000 6GB and RTX 4060 8GB still keep heavier partial-offload picks under control.

[Ramachandrajoshi]

Same for my setup, Suggested top model actually runs, but not usable due to CPU offloading and painfully slow tps

[Andyyyy64]

Thanks for the screenshot. That is useful, and I agree the top pick should not be "technically runs, but only through painful CPU offload."

A couple of fixes have landed since this issue was opened. #108 removed one ranking bug around partial offload, and #119 is tracking the next UX piece: a full-GPU-only fit filter for people who would rather avoid heavy CPU offload entirely.

Could you try the latest release and paste these outputs?

whichllm --version
whichllm hardware
whichllm --status --top 10 --json

If the top result is still mostly CPU-offloaded and too slow in the latest version, I will use your setup as another regression case for the filter/ranking work.

[Ramachandrajoshi]

here it is:

PS C:\Users\ramue> uvx whichllm@latest --version
0.5.12
PS C:\Users\ramue> uvx whichllm@latest hardware

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────── Hardware Info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ GPU 0: NVIDIA GeForce GTX 1660 SUPER — 6.0 GB (CC 7.5, CUDA 13.3) — BW: 336 GB/s │
│ GPU 1: NVIDIA GeForce RTX 5060 Ti — 15.9 GB (CC 10.0, CUDA 13.3) — BW: 448 GB/s │
│ CPU: AMD Ryzen 7 5800X 8-Core Processor — 8 cores (AVX2) │
│ RAM: 95.9 GB │
│ Disk free: 151.7 GB │
│ OS: windows │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

PS C:\Users\ramue> uvx whichllm@latest --status --top 10 --json

{
  "hardware": {
    "gpus": [
      {
        "name": "NVIDIA GeForce GTX 1660 SUPER",
        "vendor": "nvidia",
        "vram_bytes": 6442450944,
        "usable_vram_bytes": 5905580032,
        "memory_bandwidth_gbps": 336.0,
        "shared_memory": false
      },
      {
        "name": "NVIDIA GeForce RTX 5060 Ti",
        "vendor": "nvidia",
        "vram_bytes": 17103323136,
        "usable_vram_bytes": 16248156980,
        "memory_bandwidth_gbps": 448.0,
        "shared_memory": false
      }
    ],
    "cpu": "AMD Ryzen 7 5800X 8-Core Processor",
    "cpu_cores": 8,
    "ram_bytes": 102975348736,
    "ram_budget_bytes": null,
    "budget_notes": [
      "VRAM headroom: auto reserve applied per GPU"
    ],
    "os": "windows"
  },
  "models": [
    {
      "rank": 1,
      "model_id": "Qwen/Qwen3.6-27B",
      "parameter_count": 27781427952,
      "published_at": "2026-04-21T07:50:43.000Z",
      "downloads": 5965321,
      "quant_type": "Q4_K_M",
      "file_size_bytes": 15627053223,
      "vram_required_bytes": 19307400599,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 11.037269633544396,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        3.9,
        22.1
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 87.87,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 2,
      "model_id": "google/gemma-4-31B-it",
      "parameter_count": 32682372656,
      "published_at": "2026-03-11T18:22:36.000Z",
      "downloads": 11347496,
      "quant_type": "Q3_K_M",
      "file_size_bytes": 14298538037,
      "vram_required_bytes": 18442907171,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 10.966156091920181,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        3.8,
        21.9
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 83.14,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 3,
      "model_id": "Qwen/Qwen3-30B-A3B",
      "parameter_count": 30532122624,
      "published_at": "2025-04-27T03:43:05.000Z",
      "downloads": 2489551,
      "quant_type": "Q4_K_M",
      "file_size_bytes": 17174318976,
      "vram_required_bytes": 18640479744,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 102.21037037037037,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        35.8,
        204.4
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "MoE speed uses active parameters plus a bandwidth-scaled dispatch/read floor.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 81.38,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 4,
      "model_id": "google/gemma-4-26B-A4B-it",
      "parameter_count": 26544131376,
      "published_at": "2026-03-11T21:25:57.000Z",
      "downloads": 12999904,
      "quant_type": "Q4_K_M",
      "file_size_bytes": 14931073899,
      "vram_required_bytes": 16508210871,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 80.69239766081871,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        28.2,
        161.4
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "MoE speed uses active parameters plus a bandwidth-scaled dispatch/read floor.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 79.65,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 5,
      "model_id": "openai/gpt-oss-20b",
      "parameter_count": 21511953984,
      "published_at": "2025-08-04T22:33:29.000Z",
      "downloads": 6881852,
      "quant_type": "Q6_K",
      "file_size_bytes": 17478462612,
      "vram_required_bytes": 19027855533,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 53.6068376068376,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        18.8,
        107.2
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "MoE speed uses active parameters plus a bandwidth-scaled dispatch/read floor.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 78.25,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 6,
      "model_id": "zai-org/GLM-4.7-Flash",
      "parameter_count": 31221488576,
      "published_at": "2026-01-19T06:28:10.000Z",
      "downloads": 1958314,
      "quant_type": "Q3_K_M",
      "file_size_bytes": 13659401252,
      "vram_required_bytes": 16374044324,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 29.866666666666664,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        10.5,
        59.7
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "MoE speed uses active parameters plus a bandwidth-scaled dispatch/read floor.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 76.3,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "mit"
    },
    {
      "rank": 7,
      "model_id": "Qwen/Qwen3-14B",
      "parameter_count": 14768307200,
      "published_at": "2025-04-27T03:42:45.000Z",
      "downloads": 2173545,
      "quant_type": "Q6_K",
      "file_size_bytes": 11999249600,
      "vram_required_bytes": 14447513870,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 13.06748381998821,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        4.6,
        26.1
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 73.61,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 8,
      "model_id": "Qwen/QwQ-32B",
      "parameter_count": 32763876352,
      "published_at": "2025-03-05T14:16:59.000Z",
      "downloads": 73425,
      "quant_type": "Q3_K_M",
      "file_size_bytes": 14334195904,
      "vram_required_bytes": 18486281813,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 10.938876589250778,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        3.8,
        21.9
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 72.93,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 9,
      "model_id": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
      "parameter_count": 32763876352,
      "published_at": "2025-01-20T09:19:00.000Z",
      "downloads": 742830,
      "quant_type": "Q3_K_M",
      "file_size_bytes": 14334195904,
      "vram_required_bytes": 18486281813,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 10.938876589250778,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        3.8,
        21.9
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 71.96,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "mit"
    },
    {
      "rank": 10,
      "model_id": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
      "parameter_count": 24011361280,
      "published_at": "2025-06-19T17:41:13.000Z",
      "downloads": 715171,
      "quant_type": "Q4_K_M",
      "file_size_bytes": 13506390720,
      "vram_required_bytes": 16829787942,
      "vram_available_bytes": 22153737012,
      "uses_multi_gpu": true,
      "multi_gpu_effective_vram_bytes": 19358542726,
      "estimated_tok_per_sec": 12.770251029728836,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": [
        4.5,
        25.5
      ],
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "This is a synthetic GGUF estimate for an official repo, not a measured GGUF file.",
        "Multi-GPU speed depends on layer/tensor split mode, PCIe/NVLink bandwidth, and backend support; this estimate does not assume ideal scaling."
      ],
      "quality_score": 71.29,
      "benchmark_status": "direct",
      "benchmark_source": "direct",
      "benchmark_confidence": 1.0,
      "fit_type": "full_gpu",
      "can_run": true,
      "warnings": [
        "Multi-GPU fit uses a conservative layer-split budget: 18.0 GB effective from 20.6 GB raw VRAM",
        "Heterogeneous multi-GPU setup: fit assumes uneven layer placement; speed depends on backend split mode and interconnect"
      ],
      "license": "apache-2.0"
    }
  ]
}

[Andyyyy64]

Thanks, this is useful.

The latest output is a different failure mode from the first report. It is no longer picking a mostly CPU-offloaded model. The top results are full_gpu, but only by using the mixed GPU budget from the GTX 1660 SUPER and RTX 5060 Ti. That is why the speed estimate is marked low-confidence.

Could you try the top pick in the runtime you actually use and share the real tok/s, plus whether it is splitting layers across both GPUs? That will tell me whether the multi-GPU estimate is still too optimistic.

If you want to avoid the 1660 SUPER and look for safer single-GPU 5060 Ti picks, this is also worth trying:

uvx whichllm@latest --gpu "RTX 5060 Ti 16" --gpu-only --speed usable --vram-headroom 1GB

I will keep this issue open as the real-hardware regression case until we know whether the multi-GPU recommendation is actually usable.

[Ramachandrajoshi]

Yes ,With my hardware what ever the support you needed i will provide.
Also i would like to add a filters based on context size and domain like agentic_coding, creative_writing, bio_research, vision, ocr etc....
because for coding 128k context window is kind of min requirement. for other area 32k is also enough
So context window can change model picking totally. as well as domain, gpt oss and gemma models are better on writing domains,
OCR and image explanation gemma 4 E4E is better then qwen models. at same quants.

here is the output from your command.

[Ramachandrajoshi]

oh, if we put vram-headroom, it will treat both the gpus individually, but LMStudio can share the model on both, with priority order loading we can load weights on one gpu and kvcache on another gpu.

[Andyyyy64]

Thanks. I can use your setup as a real multi-GPU test case.

The context/domain filters make sense, but I want to track them separately. --context-length 128k already affects the KV estimate and ranking, while domain ranking needs a bit more design.

For this issue, I want to focus on LM Studio. You are right that LM Studio can split weights and KV cache across GPUs in a way whichllm does not model yet. Today whichllm reserves --vram-headroom per GPU, then uses a conservative multi-GPU fit estimate.

Could you share the actual LM Studio result for the recommended model: model/quant, tok/s, GPU memory used on each GPU, and any layer split or weights/KV placement shown by LM Studio?

[Ramachandrajoshi]

here is the lm studio part:

[Andyyyy64]

Thanks, this helps.

I read the screenshots as Qwen3.6 27B Q4_K_M at 131k context, 64 GPU-offloaded layers, about 17.18 GB GPU memory, 31.58 GB total memory, and about 8.29 tok/s in LM Studio.

That is within the low-confidence estimate range whichllm gave for your hardware, so I do not want to change the ranker from this result alone. The remaining gap is more specific: LM Studio can place weights and KV cache differently from the way whichllm currently explains multi-GPU fit.

I will keep this issue as the real multi-GPU behavior case. Context/domain filters should be tracked separately so this one does not become too broad.