{"full_name":"ggml-org/llama.cpp","owner":"ggml-org","name":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp","description":"LLM inference in C/C++","homepage":"https://llama.app","topics":[],"language":"C++","license":"MIT","avatar":"https://avatars.githubusercontent.com/u/134263123?v=4","created_at":"2023-03-10T18:58:00Z","pushed_at":"2026-11-10T09:23:07Z","metrics":{"stars":133886,"forks":25324,"watchers":874,"issues":10774,"prs":16942,"releases":1000,"issues_open":null,"issues_closed":null,"prs_open":null,"prs_closed":null,"prs_merged":null,"discussions":3868},"growth":{"day":{"stars":93,"forks":45,"base_day":"2026-11-09"}},"history":{"dates":["2026-11-09","2026-11-10"],"stars":[133793,133886],"forks":[25279,25324],"watchers":[874,874],"issues_open":[null,null],"prs_open":[null,null]},"boards":{"total":86},"same_owner":[{"full_name":"ggml-org/ggml","name":"ggml","boards":{}},{"full_name":"ggml-org/whisper.cpp","name":"whisper.cpp","boards":{}}],"issues":[{"number":10528,"title":"Misc. bug: Inconsistent Vulkan segfault","comments":83,"created_at":"2024-11-26T19:54:03Z"},{"number":20837,"title":"Eval bug: Qwen3.5 9B often prints tool calls in XML and stops when thinking is enabled - tool calls inside thinking block","comments":60,"created_at":"2026-03-21T11:21:18Z"},{"number":10982,"title":"Research: Performance differences between Metal (macOS) and Vulkan (Linux)","comments":58,"created_at":"2024-12-26T11:12:21Z"},{"number":14909,"title":"Feature Request: Implement missing ops from backends","comments":56,"created_at":"2025-07-28T03:39:11Z"},{"number":2164,"title":"mpi : attempt inference of 65B LLaMA on a cluster of Raspberry Pis","comments":54,"created_at":"2023-07-10T16:12:22Z"},{"number":24066,"title":"Eval bug: Vulkan: performance drop in recent builds","comments":49,"created_at":"2026-06-03T10:06:36Z"},{"number":11970,"title":"Misc. bug: The KV cache is sometimes truncated incorrectly when making v1/chat/completions API calls","comments":49,"created_at":"2025-02-20T11:20:01Z"},{"number":14702,"title":"Misc. bug: OpenAI API v1/responses llama-server","comments":41,"created_at":"2025-07-15T21:10:07Z"}],"discussions":[{"number":31895,"title":"Is llama-server secure to use from multiple users and rag from multiple users as service? Do it save the memory and data?","comments":1,"category":"Q&A","created_at":"2026-11-08T13:25:25Z"},{"number":31894,"title":"MAPLE: Pruning fine-grained MoE GGUF containers for 100% In-VRAM edge inference","comments":1,"category":"Show and tell","created_at":"2026-11-08T13:24:45Z"},{"number":31880,"title":"llama starter: A WinUI wrapper for llama-server.exe","comments":0,"category":"Show and tell","created_at":"2026-11-08T09:15:14Z"},{"number":31876,"title":"RFC: automatic placement of MoE weights on VRAM, RAM and flash, to run models bigger than memory at max prefill and decode speed","comments":0,"category":"Ideas","created_at":"2026-11-08T08:00:14Z"},{"number":31869,"title":"[Showcase] MAPLE: Sub-2-BPW Pruning of Gemma 4 26B-A4B for 100% In-VRAM GPU Execution on 6GB VRAM (45.4 TPS)","comments":0,"category":"Show and tell","created_at":"2026-11-08T04:05:16Z"},{"number":31868,"title":"Windows (CUDA) build available: prebuilt zip and launchers","comments":0,"category":"Show and tell","created_at":"2026-11-08T03:57:04Z"},{"number":31858,"title":"`llama tune` - finding the best config for your setup","comments":2,"category":"Ideas","created_at":"2026-11-07T18:07:28Z"},{"number":31846,"title":"ArtiGram: a free gallery of interactive HTML artifacts with an MCP server","comments":0,"category":"Show and tell","created_at":"2026-11-07T08:15:25Z"}],"latest_release":{"name":"v0.9.0","tag":"v0.9.0","published_at":"2026-11-04T13:08:06Z","body":"## Overview\r\n\r\nThis release adds OCP FP8 E4M3 quantization support, a new `flash_attn_ext_rows` op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. `ggml` is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.\r\n\r\n### Highlights\r\n\r\n- Quant: OCP FP8 E4M3 tensor type (saturating mode) with conversion helpers and a new ftype ([#27512](https://github.com/ggml-org/llama.cpp/pull/27512))\r\n- llama: `flash_attn_ext_rows` - more efficient unified KV for multi-seq, restoring non-unified KV prefill performance ([#29510](https://github.com/ggml-org/llama.cpp/pull/29510))\r\n- ggml: batched tensor copy API (`ggml_backend_tensor_copy_n`/`set_n`/`get_n`) superseding the `_2d` API, implemented for Metal and CUDA ([#30881](https://github.com/ggml-org/llama.cpp/pull/30881))\r\n\r\n- For a 3rd year in a row, `llama.cpp` is a featured repo in the [Github Octoverse report](https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/):\r\n\r\n  <img width=\"1920\" height=\"1080\" alt=\"image\" src=\"https://github.com/user-attachments/assets/1ac82af7-576f-487b-bba3-2b56bb641022\" />\r\n\r\n\r\n### API changes\r\n\r\n- llama.h: add `LLAMA_FTYPE_MOSTLY_F8_E4M3` ([#27512](https://github.com/ggml-org/llama.cpp/pull/27512))\r\n- mtmd.h: add `MTMD_GEN_AUDIO_TYPE_CHATTERBOX` and `n_voices` to `mtmd_gen_audio_info` ([#30993](https://github.com/ggml-org/llama.cpp/pull/30993))\r\n- mtmd.h: add predefined-voice APIs: `mtmd_gen_audio_voice_name()`, `mtmd_gen_audio_get_voice()`, `mtmd_gen_state_set_ref()` ([#30993](https://github.com/ggml-org/llama.cpp/pull/30993))\r\n- mtmd.h: add `mtmd_bitmap_is_voice()` and `mtmd_gen_audio_read_tensor()` ([#30993](https://github.com/ggml-org/llama.cpp/pull/30993))\r\n- mtmd.h: add `MTMD_VIDEO_TEMPLATE_QWEN3VL` video prompt template ([#31619](https://github.com/ggml-org/llama.cpp/pull/31619))\r\n- server: `GET /props` reports the model's predefined TTS `voices`; the TTS endpoint accepts predefined voice names ([#30993](https://github.com/ggml-org/llama.cpp/pull/30993))\r\n- server: `/v1/responses` now merges `extra_params` (seed, min_p, ...) into the converted chat-completions response ([#30959](https://github.com/ggml-org/llama.cpp/pull/30959))\r\n\r\n### New models\r\n\r\n- Clover-35B-A3B-Preview diffusion model, AR mode ([#31306](https://github.com/ggml-org/llama.cpp/pull/31306))\r\n- Limite 1B - Violetto ([#29433](https://github.com/ggml-org/llama.cpp/pull/29433))\r\n- Metis-1.6 (Praxis ~2.9B and Logos ~10.2B): hybrid Mamba-2 / GQA attention with fixed-depth recurrence and LatentMoE ([#31134](https://github.com/ggml-org/llama.cpp/pull/31134))\r\n- Mistral Large 4 ([#31376](https://github.com/ggml-org/llama.cpp/pull/31376))\r\n\r\n### Core changes\r\n\r\n- gemma4: graph optimizations with massive prefill gains for E2B and E4B ([#31356](https://github.com/ggml-org/llama.cpp/pull/31356)); create `token_embd` before the tied output in gemma3n and gemma4 ([#31394](https://github.com/ggml-org/llama.cpp/pull/31394))\r\n- LFM3: enable NVFP4 GGUFs ([#31608](https://github.com/ggml-org/llama.cpp/pull/31608)), enable tensor split ([#31431](https://github.com/ggml-org/llama.cpp/pull/31431)), quantize the audio codec decoder ([#31386](https://github.com/ggml-org/llama.cpp/pull/31386))\r\n- spec dec: save and restore MTP hidden state in context checkpoints ([#30141](https://github.com/ggml-org/llama.cpp/pull/30141)) and record LFM3 layer inputs ([#31598](https://github.com/ggml-org/llama.cpp/pull/31598))\r\n- dfla"},"readme":true,"readme_zh":false,"news":[{"title":"Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI","url":"https://github.com/ggml-org/llama.cpp/discussions/19759","hn":"https://news.ycombinator.com/item?id=47088037","source":"Hacker News","lang":"en","date":"2026-02-20","points":839,"comments":223},{"title":"Vision Now Available in Llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md","hn":"https://news.ycombinator.com/item?id=43943047","source":"Hacker News","lang":"en","date":"2025-05-10","points":550,"comments":104},{"title":"Mistral Integration Improved in Llama.cpp","url":"https://github.com/ggml-org/llama.cpp/pull/14737","hn":"https://news.ycombinator.com/item?id=44862583","source":"Hacker News","lang":"en","date":"2025-08-11","points":95,"comments":15},{"title":"Llama.cpp: Add GPT-OSS","url":"https://github.com/ggml-org/llama.cpp/pull/15091","hn":"https://news.ycombinator.com/item?id=44799218","source":"Hacker News","lang":"en","date":"2025-08-05","points":35,"comments":0},{"title":"LLama.cpp Got Screwd","url":"https://github.com/ggml-org/llama.cpp/discussions/25482","hn":"https://news.ycombinator.com/item?id=48844268","source":"Hacker News","lang":"en","date":"2026-07-09","points":11,"comments":5},{"title":"Tell HN: Llamacpp now supports unified system RAM offloading on Linux","url":"https://news.ycombinator.com/item?id=47510953","hn":"https://news.ycombinator.com/item?id=47510953","source":"Hacker News","lang":"en","date":"2026-03-24","points":6,"comments":0},{"title":"Yet another reminder why you should not use Ollama","url":"https://github.com/ggml-org/llama.cpp/pull/19324","hn":"https://news.ycombinator.com/item?id=46895014","source":"Hacker News","lang":"en","date":"2026-02-05","points":6,"comments":5},{"title":"Is GitHub Releases Down?","url":"https://news.ycombinator.com/item?id=44303056","hn":"https://news.ycombinator.com/item?id=44303056","source":"Hacker News","lang":"en","date":"2025-06-17","points":5,"comments":3},{"title":"What's easier in llama.cpp v0.8.0? Finding models, long conversations, and things to watch out for after upgrading","url":"https://note.com/ai_gyaru_lab/n/n98a992874ad4?hl=en","source":"note","lang":"en","date":"2026-10-29"},{"title":"Build llama.cpp with DFlash 2 support (PR #27342): git clone","url":"https://x.com/analogalok/status/2092181751363936543","source":"x","lang":"en","date":"2026-08-25"},{"title":"Accelerate llama.cpp on AMD EPYC: ZenDNN Delivers Up to 4.5x the Prompt Processing","url":"https://www.amd.com/en/developer/resources/technical-articles/2026/llama-cpp-on-amd-epyc.html","source":"amd","lang":"en","date":"2026-08-18"},{"title":"I ditched LM Studio for llama.cpp, and my local LLM doesn't feel like a downgrade anymore","url":"https://www.xda-developers.com/ditched-lm-studio-for-llama-cpp-and-local-llm-doesnt-feel-like-downgrade-anymore/","source":"XDA Developers","lang":"en","date":"2026-05-23"},{"title":"Free software 'llama.cpp' that can run various AI models locally supports multimodal input and enables image explanations, etc.","url":"https://gigazine.net/gsc_news/en/20250512-llama-cpp--multimodal-vision-support/","source":"GIGAZINE","lang":"en","date":"2025-05-12"},{"title":"Intel announces that 'Run llama.cpp Portable Zip on Intel GPU with IPEX-LLM', which allows various AIs to run on a local Windows PC, also supports DeepSeek","url":"https://gigazine.net/gsc_news/en/20250309-run-llama-portable-zip","source":"GIGAZINE","lang":"en","date":"2025-03-09"},{"title":"``llama.cpp'', which made Meta's large-scale language model ``LLaMA'' executable on a Mac with M1, can be executed with less than 6 GB of memory usage by updating","url":"https://gigazine.net/gsc_news/en/20230403-llama-cpp-ram","source":"GIGAZINE","lang":"en","date":"2023-04-03"},{"title":"BaseRT: 在 Apple 芯片上通过原生 Metal 实现同类最佳的 LLM 推理","url":"https://news.google.com/rss/articles/CBMiVEFVX3lxTFB2R0ZVQldrYkVVSVdoN21iQUZHYnRJeGpBVG1IVlNrbTF0OGtzV0R0LVQ0QVJQRHhEbkZFdHhBM3Y5YnJkWWI0aHg3TFY2TTBQY0hYaw?oc=5","source":"alphaXiv","lang":"zh","date":"2026-09-02"},{"title":"Run Muse Glimmer for Local Vibe Coding with llama.cpp, DFlash, and Pi","url":"https://news.google.com/rss/articles/CBMimAFBVV95cUxOaEowQkR4YTljaVN1bUk1Z1BNTlRWNzVPY2xTa21Wc3d6S21XVVdyWjdYMDFBdGdhTVlONHVGc0x3TGhUM2M3LUlaWG8wVmdrenJjVk4tWDZDRGZDYUFfajRISF9ncjFqLVpBODl5YWxTbWN1UFVpclBsTmlLOVozaWVmVy1mM2xUN0pQVkNoWE5vclFabmdGcg?oc=5","source":"KDnuggets","lang":"en","date":"2026-08-21"},{"title":"Windows本地AI升级，llama.cpp支持多后端，一键跑GGUF无审查模型","url":"https://news.google.com/rss/articles/CBMiW0FVX3lxTE5tSTkyR0hoYXVZME5VcHV0cXpHWnRLS1NqR2Q0cGVITDZYSGx0VTh5d21VdHRUb2ozRUxkM09OTThvWm9SUUVMUktDSHVqMEd5dnEtd2hNNDdrdTg?oc=5","source":"blog.upx8.com","lang":"zh","date":"2026-04-08"},{"title":"Running LLMs on Raspberry Pi and Edge Devices: A Practical Guide","url":"https://news.google.com/rss/articles/CBMiXkFVX3lxTFBpVnVtRTVaYkV1TEFuNFV5aW1LaV9kYmFhUlBnS3RudHcxOHA4ekc1d1lhTG9QVXpFdFVSaWUyU2s0MGZqZ09CbFJLaEMzMTd5NjAxblF6T0lPaUl2TUE?oc=5","source":"SitePoint","lang":"en","date":"2026-02-25"},{"title":"GGML Joins Hugging Face: What This Means for Local Model Optimization","url":"https://news.google.com/rss/articles/CBMimwFBVV95cUxNc0VZdEE0SnlBTnF6bERlVjNXWV9mUW1Va2pVODU3X3VsN0JkdjQ1bnRsczkySnl0MkxHY0J2SE1tS21GLUVsMnFNcG85TWRhbTVPbHkwMktIcU1HZFhiSU5RR2lNOUxaNHhwSFpaeFUzMXBpTjh3cDh4QldrQUI4Wm43ZXN3NDVBT2NMZlllVUF0SUJzTEpkZkg3NA?oc=5","source":"SitePoint","lang":"en","date":"2026-02-22"},{"title":"How to run LLMs on PC at home using Llama.cpp","url":"https://news.google.com/rss/articles/CBMioAFBVV95cUxPbHh2dFU3bDh6d0NoOW5rYkFnblhFVHhNMl8zR3FyUjU1dDlGcTdZWml5b1R3dDUtdi1EZXB1Y2x1MF81M3N2YmZ3Q1JTWjJ5eUxVMmFpZVpXandtYUNybURTbXpsS3o5SV9pS3ZCU09vMmJ2MVhXX2JjMWVqMUNxcWhTbGVweHdTdHZvc0c4OGJTbklVaUlCRTFSR04yTFhP?oc=5","source":"The Register","lang":"en","date":"2025-08-24"},{"title":"在llama.cpp中为高通Adreno GPU引入全新的OpenCLTM GPU后端","url":"https://news.google.com/rss/articles/CBMiXkFVX3lxTE1DcmJWMGplVXdCVDA0b0xSVG1iQWY3M1ZIRzN1bkc3NDUzY2QyTVhOVWV5eWVUdEZCcWJtX0NlX1ZhaWpDeGZBekhZYzNOcXBPMTBqc0Rsc2xuZlJDelE?oc=5","source":"CSDN","lang":"zh","date":"2024-12-11"},{"title":"Optimizing llama.cpp AI Inference with CUDA Graphs","url":"https://news.google.com/rss/articles/CBMijgFBVV95cUxOUldOYmNTRXlDM01naE4yVnNuS01Hc3B0VS04SE5HaXJyUTlQREhtLWg4VVE0Um9aWHdfMmZtNUo0UG8xWkNBNi1BX2dXYThBYUlVLU5zREgxdHoyaXhHWG9MSkhyZ3BoSjdkU294WTFDSkt2UU9tTmNMc1BnM1pvVlZuaXBWZ2pWWEI3QThn?oc=5","source":"NVIDIA Developer","lang":"en","date":"2024-08-07"},{"title":"研究完llama.cpp，我发现手机跑大模型竟这么简单","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTE5lUUdFeUo2bHZWMDBrTngyN1QydUo4eFpIcExBWlNOTGllT2Z3TXJsT3lQbzRoM3I1UDF0NUxHaUd3bzBoQ0Zfbw?oc=5","source":"品玩","lang":"zh","date":"2023-08-17"},{"title":"Llama 2本地运行指南和工具Ollama，已4k星","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTFBxVkJDbi1NemxMWHdOZ2hQazAtNS1kNkQ0TGVjbWZXcWRXT0ZGeHd4aUhoeUhMWWlHN2xmbGVZeTJjOWZobWcycQ?oc=5","source":"智源社区","lang":"zh","date":"2023-08-03"}],"stale":false,"snapshot_at":"2026-11-10T09:35:25Z","history_start":"2026-09-06"}