README

DwarfStar aims to be the best way to run a few excellent large
language models on consumer hardware (that is, hardware that people
can actually own). To reach this goal, we are building
a small native inference engine optimized first for
DeepSeek V4 Flash (including the experimental vision model),
DeepSeek V4.1 Flash (Metal, and text inference on CUDA),
and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and
DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA). The code is self-contained and
deliberately narrow, not a general GGUF runner: you need to use the
GGUF files the project produces, that are part of the project
itself.

We test things in integration: model loading, prompt rendering,
tool calls, KV state, the HTTP server, and the coding agent are built and tested together.
The repository also includes tools and data for GGUF, imatrix, quality, and speed.

 Supported hardware

 Metal, the primary target, on Macs with 64 GB (for Qwen Flash next) and 96 GB or more (for DeepSeek and GLM). Smaller machines can use SSD streaming. SSD streaming is also needed in order to run very large models such as full GLM 5.x (not Flash) on 128GB systems.
 NVIDIA CUDA, the DGX Spark is our main gaol. DwarfStar also supports multi-GPU systems that are not supported by other backends, for instance it can run DeepSeek v4 Flash on Ada Lovelace cards.
 ROCm on Strix Halo systems such as the Framework Desktop.

This project would not exist without llama.cpp and GGML, make sure to read
the acknowledgements section, a big thank you to Georgi Gerganov and all the
other contributors.

Model support is intentionally opportunistic. The project follows the best open
weights for useful local machine sizes, especially 128 GB laptops and 256/512 GB
workstations. A model may be removed when a better replacement arrives.

 So, what can I do with this software?

 You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.
 You can use multiple CUDA cards as a multi-user LLM server. Ada Lovelace, including L40S, is supported: newer models can run here even when their other inference implementations require newer GPUs. Our eight-L40S Flash setup has reached about 126 t/s aggregate generation with 16 sessions.
 Using two 128 GB Macs connected with RDMA, you can run 4-bit DeepSeek Flash or GLM 5.3 Flash with tensor parallelism. Larger GLM 5.2 quants need larger machines, such as Mac Studios.
 You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.

 Motivations

 Capable open-weight models now fit on high-end personal machines.
 DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.
 Compressed KV caches and fast local SSDs make long contexts practical.
 The idea of an inference system specialized for a few models.

 AI full disclosure

 This software is devel