README

Bonsai Demo

Backend and model format compatibility: BACKEND-SUPPORT.md.


  



  Website  | 
  GitHub  | 
  Discord



  Models:
  Bonsai 2 27B GGUF ·
  Bonsai 2 27B MLX


---

Run Bonsai 2 2B, 4B, 9B, and 27B locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU.

For the earlier Bonsai 1-bit and Ternary-Bonsai models (27B, 8B, 4B, 1.7B), see
Bonsai1_README.md. For troubleshooting, see FAQ.md.

 🌱 Bonsai 2: 2B, 4B, 9B, and 27B

Bonsai 2 now includes 2B, 4B, and 9B models alongside 27B. Pick a smaller model
for a lighter local setup; 27B remains the demo's default. GGUF and MLX checkpoints
are available for each size.

The following benchmark and memory highlights describe 27B; see each smaller
model's card for its specifications:

- 98.2% of FP16 intelligence retained at roughly a ninth of the size, with the reasoning core
  intact: math within half a point of full precision, coding level with the baseline.
- Vision: send photos, screenshots and PDFs and ask about them, on both llama.cpp and MLX
  (see VISION.md).
- Agentic tool calling: native OpenAI-style tool_calls with full round-trips, plus MCP servers
  in both demo UIs (see TOOLS.md).
- Thinking: a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.
- 262K-token context, kept practical on-device by the hybrid-attention backbone.
- 1.75 bits per weight in the PTQ1_0 packing. A second packing, PQ2_0, trades 1.3 GB for
  faster prompt processing and is what this demo downloads by default.
  See MODEL-FORMATS.md.

Official Q2_0 works with upstream llama.cpp b11556 or newer.
PQ2_0 and PTQ1_0 still require the PrismML fork.
This demo downloads PQ2_0 and the matching fork binaries by default.
All three packings contain the same model weights, with different memory/speed trade-offs.

Quick Start below gets you there in two commands: ./setup.sh downloads Bonsai 2 27B, then
./scripts/start_llama_server.sh gives you chat, vision and tools at http://localhost:8080.

 Choose a smaller model

bash
export BONSAI_MODEL=4B     2B, 4B, 9B, or 27B
./setup.sh
./scripts/start_llama_server.sh


On Windows, set $env:BONSAI_MODEL = "4B" before setup and launch. For Apple Silicon,
use ./scripts/run_mlx.sh or ./scripts/start_mlx_server.sh with the same model selection.
See small-model sampling and API examples: the 2B/4B/9B profiles
use min-p 0 and a 64-token presence-penalty window. The 27B settings below are separate.

 Best Practices (Bonsai 2 27B)

These match the model card. The start scripts already apply the thinking-mode sampling, so you only need these when calling the model from your own client.

 Generation Parameters

| | Thinking mode (default) | Instruct / non-thinking |
|---|---|---|
| temperature | 1.0 | 0.7 |
| top_p | 0.95 | 0.80 |
| top_k | 20 | 20 |
| min_p | 0.05 | 0.0 |
| presence_penalty | 0.0 | 1.5 |
| repetition_penalty | 1.0 | 1.0 |

min_p=0.05 drops tokens far less likely than the top choice; in our tests it scored at