README

whichllm








  


The smartest model your PC can actually run, including ones you didn't know it could.

whichllm reads your GPU, RAM and memory bandwidth, works out which open
models fit and how fast each one will generate, and ranks them by benchmark
results. Then it prints the command that runs the top pick the way it was
estimated.

日本語版はこちら

 Quick start

bash
uvx whichllm@latest


Shopping for a GPU? Simulate one:

bash
uvx whichllm@latest --gpu "RTX 4090"


To keep it around:

bash
uv tool install whichllm
brew install andyyyy64/whichllm/whichllm
pip install whichllm


 Example

A desktop with an 8 GB RTX 4060 and 32 GB of DDR4 (output trimmed):

text
$ whichllm

   Model                         Quant   Fit       Speed     Score
 1  Qwen/Qwen3.6-35B-A3B          Q6_K    GPU+RAM   16 tok/s   68.8
 2  unsloth/gemma-4-12b-it-GGUF   IQ4_XS  Full GPU  37 tok/s   68.1
 3  google/gemma-4-26B-A4B-it     Q4_K_M  GPU+RAM   11 tok/s   64.5
 4  Qwen/Qwen3.5-9B               Q5_K_M  Full GPU  36 tok/s   64.2
 5  zai-org/GLM-4.7-Flash         Q5_K_M  GPU+RAM   15 tok/s   62.3

  1 Qwen3.6-35B-A3B also stands in for older generations of its line:
  Qwen3.5-35B-A3B, Qwen3-30B-A3B, Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF

  Run 1 with llama.cpp:
    llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q6_K -c 4096 -ngl 99 --n-cpu-moe 35 -fa on


The top pick is a 35B model on an 8 GB card. It is a mixture-of-experts
model, so each token only uses about 3B of its parameters. whichllm keeps
attention, the shared weights and the KV cache in VRAM and leaves the experts
of 35 of its 40 layers in system RAM, which is what llama.cpp's --n-cpu-moe
does. The speed estimate counts only what each token reads from each place.
A fits-in-VRAM check would have skipped the model, and treating it as a
plain partial offload would have rated it far too slow.

On a 24 GB RTX 4090 the same logic picks Qwen3.8-27B, although Qwen3-32B
also fits there at Q4_K_M. The 27B is the newer generation of the same model
line at about the same size and scores higher on current benchmarks, so the
32B is not listed separately; the note under the table names it.

Speeds are estimates from memory bandwidth, not measurements of your
machine. The table marks how confident each estimate is.

Here is one checked on real hardware. On a 16 GB RTX 5080 with DDR5-5600,
whichllm estimates 70 tok/s for Qwen3.6-35B-A3B at Q4_K_M, with this command:

bash
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M -c 4096 -ngl 99 --n-cpu-moe 15 -fa on


It measured 79 tok/s on llama.cpp b11977, from a 22 GB file on a 16 GB card.
whichllm bench runs the same comparison on your machine.

 How it ranks

Quality comes from Artificial Analysis and LiveBench, put on one scale and
averaged. --profile coding, math and vision rank on benchmarks for that
task instead, such as SciCode and LiveCodeBench for coding. The Open
LLM Leaderboard and Chatbot Arena are frozen, so they only score models
nothing current covers, and older