{"full_name":"JustVugg/colibri","owner":"JustVugg","name":"colibri","url":"https://github.com/JustVugg/colibri","description":"A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own.","homepage":"https://justvugg.github.io/colibri","topics":[],"language":"C","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/13022503?u=16b428ea8d9b8897f7268ab4257b57bfbad64c45&v=4","created_at":"2026-07-01T12:27:49Z","pushed_at":"2026-11-10T08:23:46Z","metrics":{"stars":49551,"forks":5833,"watchers":393,"issues":630,"prs":1999,"releases":21,"issues_open":null,"issues_closed":null,"prs_open":null,"prs_closed":null,"prs_merged":null,"discussions":71},"growth":{"day":{"stars":353,"forks":61,"base_day":"2026-11-09"},"week":{"stars":3132,"forks":462,"base_day":"2026-11-03"},"month":{"stars":9818,"forks":1474,"base_day":"2026-10-11"}},"history":{"dates":["2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-09","2026-11-10"],"stars":[27415,27695,28439,31724,33520,34853,35640,36113,36361,36793,37102,37321,37505,37633,37756,38112,38386,38826,39023,39273,39733,39971,40272,40527,40764,40976,41474,41712,41905,42138,42380,42571,42859,43034,43208,43352,43573,43705,44039,44192,44639,44925,45170,46419,47136,49198,49551],"forks":[3001,3026,3108,3361,3521,3661,3743,3809,3854,3936,4001,4030,4058,4076,4090,4156,4193,4249,4271,4303,4359,4399,4430,4450,4478,4499,4580,4612,4638,4690,4728,4751,4801,4826,4853,4875,4916,4952,5005,5029,5065,5115,5150,5371,5452,5772,5833],"watchers":[234,235,248,268,269,277,282,286,289,287,291,292,291,293,295,301,308,315,318,323,322,321,322,323,323,323,327,331,333,338,339,340,339,342,347,348,346,343,348,354,353,355,357,376,379,393,393],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]},"boards":{"daily":99,"weekly":37,"monthly":58,"weekly_rate":62,"monthly_rate":84},"same_owner":[],"issues":[{"number":1405,"title":"[Feature]: Provide the necessary Windows coli_cuda.dll files or update instructions to build them","comments":20,"created_at":"2026-09-09T23:05:42Z"},{"number":594,"title":"[Feature]: Lossless entropy compression for streamed expert traffic (transport layer, any container)","comments":18,"created_at":"2026-07-24T17:01:45Z"},{"number":1852,"title":"[Feature]: Deb or pkg","comments":16,"created_at":"2026-10-03T08:13:37Z"},{"number":708,"title":"[Experiment] expert-transition-history placement policy vs gate-momentum — controlled A/B for hypothesis #1","comments":12,"created_at":"2026-07-30T01:25:44Z"},{"number":441,"title":"[Performance]: PILOT prefetch runs at NVMe queue-depth 1, and DIRECT=1 re-reads WILLNEED-prefetched pages","comments":12,"created_at":"2026-07-19T20:56:28Z"},{"number":689,"title":"[Bug]: deep speculative verify batches are not token-exact vs CPU on CUDA (near-tie flips + lower draft acceptance)","comments":11,"created_at":"2026-07-29T16:14:50Z"},{"number":2583,"title":"[Bug]: qwen36 Vulkan degradation when all experts are resident.","comments":10,"created_at":"2026-11-04T08:27:01Z"},{"number":2541,"title":"[Feature]: can glm5.3? windows gpu nvidia release dll?","comments":9,"created_at":"2026-11-01T21:57:03Z"}],"discussions":[{"number":2649,"title":"AVX2 Not detected on a CPU with AVX2 (E5-2697A V4 Xeon)","comments":1,"category":"Q&A","created_at":"2026-11-06T13:10:34Z"},{"number":2644,"title":"Checking the internet for information.","comments":1,"category":"Q&A","created_at":"2026-11-06T08:03:15Z"},{"number":2553,"title":"PS5 port?","comments":0,"category":"Ideas","created_at":"2026-11-02T15:35:09Z"},{"number":2529,"title":"Interesting new models","comments":3,"category":"General","created_at":"2026-10-31T13:37:53Z"},{"number":2496,"title":"Abliterated models","comments":2,"category":"Ideas","created_at":"2026-10-30T06:27:09Z"},{"number":2455,"title":"Colibri Benchmarking and Optimization Report","comments":0,"category":"Show and tell","created_at":"2026-10-28T20:44:29Z"},{"number":2409,"title":"will the use of colibri destroy the SSD?","comments":1,"category":"Q&A","created_at":"2026-10-26T11:52:32Z"},{"number":2291,"title":"Muse Spark Open (AI never sleeps 😭)","comments":1,"category":"General","created_at":"2026-10-22T09:50:29Z"}],"latest_release":{"name":"colibri v2.1.0","tag":"v2.1.0","published_at":"2026-10-31T12:50:50Z","body":"**Huge open models on the machine you already have**: on the CPU, on any GPU through Vulkan, on NVIDIA cards through CUDA. Pure C, one file per model family.\n\n387 pull requests since v2.0.0, 208 of them from 35 contributors. GGUF files now run natively on every engine, read exactly as published. Four new models arrive, Mistral-Large-4 (1T) among them. CUDA gets device chains of its own and a routed-expert tier for every engine. Qwen3.8-Flash-Next runs a 256k context at the speed of 64k, an agent's turns read only what is new, and there is a new chat UI.\n\n<p align=\"center\">\n  <img src=\"https://raw.githubusercontent.com/JustVugg/colibri/v2.1.0/docs/media/ui/ui-experts.png\" width=\"900\" alt=\"the new chat UI with the expert map open\">\n</p>\n\n## Highlights\n\n- **GGUF on every engine.** `coli import` takes a GGUF where it lies and every engine runs it as published: no conversion, no second quantization, from IQ1_S to Q8_0, F16, BF16 and MXFP4, the same blocks and the same tokens on the CPU, Vulkan and CUDA. unsloth's Dynamic UD quants run as they are ([docs/gguf.md](https://github.com/JustVugg/colibri/blob/v2.1.0/docs/gguf.md)).\n- **New models.** Mistral-Large-4 (1T, 52B active: text, pictures, reasoning, tools), Muse Spark Open (816B), Flint (40B, 2.5B active), LFM3-VL-2B with its vision tower, and the published DeepSeek V4.1 Flash and Kimi K3 GGUFs.\n- **CUDA.** Every layer of Qwen3.6, Qwen3.8 and OLMoE as one chain on the card, a routed-expert tier on the card for every engine, the dense matrices on the card for DeepSeek V4, V4.1, Kimi K3, GLM-5.3 and MiMo, and a GGUF's blocks on CUDA with the prompt on the tensor cores.\n- **Vulkan.** A model larger than the card keeps the experts the card cannot hold in a host pool in RAM, and the CPU computes them inside the device's frame while the card computes its own. Prompts on the matrix units, flash attention with windows and sinks, float16 KV mirrors.\n- **Long contexts and agents.** Qwen3.8-Flash-Next at 262,144 tokens: decode 9.6 to 45.7 tok/s, prompt 15.6 to 149. A short request between two turns (a title, a summary) parks the conversation instead of taking its state, on every engine: a Hermes Agent turn went from 86 s to 4.9 s.\n- **A new chat UI.** `coli ui`: reasoning in its own block, pictures, web search, a live map of the experts as they fire, a monitor, your models, and coding agents already pointed at colibri. Seven languages.\n- **Speculative decoding.** Qwen3.6's MTP head on every backend (`Q36_MTP=1`): decode 99.7-104.5 to 115.8-120.1 tok/s on CUDA, 83.2-83.8 to 99.9-109.3 on Vulkan; DeepSeek V4.1's DSpark head drafts as deep as its confidence says, behind a gate that skips the drafts that cannot pay.\n- **Memory.** KV caches in float16: GLM-5.3 at 131,072 tokens in 11.9 GB instead of 23.9.\n\n## Measured on one 24 GB card\n\nAn i5-13500 with 62 GB of RAM and an RTX PRO 4000 Blackwell SFF (24 GB, 70 W), each model served by `coli serve`: a fresh 450-token prompt, then a 128-token streamed answer, the request's HTTP included; the better of two runs, tok/s.\n\n| GGUF | CUDA prompt | CUDA decode | Vulkan prompt | Vulkan decode | CPU prompt | CPU decode |\n|---|---:|---:|---:|---:|---:|---:|\n| Qwen3.6-35B-A3B UD-Q4_K_M | 3,491 | 104.8 | 2,076 | 92.9 | 122.6 | 13.9 |\n| LFM3-VL-2B Q4_K_M | 12,317 | 235.1 | 8,174 | 214.1 | 286.5 | 34.5 |\n| OLMoE-1B-7B-0125-Instruct Q4_K_M | 10,418 | 354.6 | 6,867 | 299.4 | 279.9 | 47.4 |\n| LFM3-VL-2B F16 | 11,524 | 103.4 | 7,767 | 99.3 | 49.7 | 6.4 |\n| Qwen3.8-Flash-Next IQ2_XS (68 GB: the experts the card cannot hold stay in RAM) | 217.1 | 51.4 | 214.3 | 50.6 | 38.3 | 9.3 |\n\n## Bigger than the RAM, on the same machine\n\nEach model from its published GGUF; what neither the card nor the RAM holds comes from two NVMe drives. Tokens per second, from each model's page in `docs/`.\n\n| GGUF | size | backend | prompt | decode |\n|---|---:|---|---:|---:|\n| Inkling-Small UD-IQ2_XXS | 77 GB | Vulkan | 15.1-33.1 | 10.92-11.53 |\n| MiMo-V2.6-Flash Q2_K_S | 106 GB | Vulkan | 8."},"readme":true,"readme_zh":true,"news":[{"title":"Show HN: Getting GLM 5.2 running on my slow computer","url":"https://github.com/JustVugg/colibri","hn":"https://news.ycombinator.com/item?id=48842459","source":"Hacker News","lang":"en","date":"2026-07-09","points":937,"comments":240},{"title":"Brio mode in Colibri: Scoring a closed set instead of generating","url":"https://github.com/JustVugg/colibri/blob/v1.12.0/docs/brio.md","hn":"https://news.ycombinator.com/item?id=49785774","source":"Hacker News","lang":"en","date":"2026-09-21","points":1,"comments":1},{"title":"Call for probes: building the GLM-5.2 Expert Atlas","url":"https://github.com/JustVugg/colibri/issues/175","hn":"https://news.ycombinator.com/item?id=49257324","source":"Hacker News","lang":"en","date":"2026-08-11","points":1,"comments":0},{"title":"JustVugg on X: \"colibri is #1 on @github today! What a milestone. Thank you to every developer, contributor and user who helped us get here. This is your achievement too.\"","url":"https://x.com/justvugg/status/2099240780619509937","source":"x","lang":"en","date":"2026-09-15"},{"title":"Colibri: The Revolutionary AI Engine Running 744B-Parameter Models on Just 25GB RAM","url":"https://www.alphamatch.ai/blog/colibri-ai-engine-glm-5-2-25gb-ram-2026","source":"alphamatch.ai","lang":"en","date":"2026-07-10"},{"title":"A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744 billion parameters, on a regular PC with 25GB of memory.","url":"https://gigazine.net/gsc_news/en/20260710-colibri-glm","source":"GIGAZINE","lang":"en","date":"2026-07-10"},{"title":"7440億パラメーターの巨大AI「GLM-5.2」をメモリ25GBの普通のPCで動かす推論エンジン「Colibrì」が登場","url":"https://gigazine.net/news/20260710-colibri-glm/","source":"GIGAZINE","lang":"zh","date":"2026-07-10"},{"title":"笔记本跑7000亿参数GLM！无GPU也行? SSD当显存用火爆GitHub","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTE56b2JoZkNLQ1JTN3NZdUx1SFJTYzR5V2tlal8yQm5ZYmZPNVV3am4xRGprRzRaV19PLTV3eVkwSWRxaW0yenJ0Sw?oc=5","source":"智源社区","lang":"zh","date":"2026-09-27"},{"title":"What repos are trending on GitHub?","url":"https://news.google.com/rss/articles/CBMifkFVX3lxTE1SNElkQXBUUWlBZFBIRFoyclZjVldGUllHYXVOSl8xT09EQVozbEdsVWtjeFgtckc4ZXhjQThVUHB3VEZTcnlYRHdXYTAxYS1EM2FnU2FrdVU0MnJ6MlhCelo0cEN6MHl6Z3FUZ1kxSVNqa005cWwybzk4ZzI0QQ?oc=5","source":"TechTarget","lang":"en","date":"2026-09-21"},{"title":"Colibrì：GLM-5.2 的 7440 亿参数模型在 25 GB 内存上运行","url":"https://news.google.com/rss/articles/CBMigAFBVV95cUxNS0tRSl9ZZXBheERhWHZwTTQycW9ueG9yajgtOEp6bmtxSEtHWUZBZmtQRUI3bFBnUDV2UUxUdjhYSDZiWXM5cjhiZEFwekZ6ZXNVeXNvZ0prZDdERTNzTUd6VTdrU0daczdRTGtfVzBBRUhLTlE0RTlZT3NSUUt1cQ?oc=5","source":"Pasquale Pillitteri","lang":"zh","date":"2026-09-13"},{"title":"磁盘流式加载让千亿参数MoE模型在笔记本上运行","url":"https://news.google.com/rss/articles/CBMigAFBVV95cUxNcW1Cc3NrRnJfMXh6RkhuUGJ2aFNxcWcySGdNVEFzOGM4bmZpTEIzMHpGNHVUbE1CV0lpeS1ncWR1TXlCdmU0ZFVZU1RZWWFjbWIxWE1ZcnNxNlZiNDc3WVRMb1dRQjlsR3RvbGtJekdLaVdHbk8yQ0VCTFdRa2RIdA?oc=5","source":"Pasquale Pillitteri","lang":"zh","date":"2026-09-13"},{"title":"Colibrì: GLM-5.2, a 744-Billion-Parameter Model on 25 GB of RAM","url":"https://news.google.com/rss/articles/CBMigAFBVV95cUxNeE0yU0ZJQzVvcHRpbFRuTHhxVUhZX1RlTFdCNGN4dFNWMjB6TW9iRHJtTTNaR1pCSVJiX3pZaFF0b1JCMkZQd3YwdjhYOURUNW44NzhtZXo2bjFpaFVfbUk2c1hmdkotV2FocjBZYVRyMVdsRzF0MVJJY1A4RnlIZg?oc=5","source":"Pasquale Pillitteri","lang":"en","date":"2026-07-12"},{"title":"Colibrì proof-of-concept gains frontier-level 1.5-TB AI model — novel approach runs on only 25GB of RAM and shows promise for local AI setups","url":"https://news.google.com/rss/articles/CBMirwJBVV95cUxNbmdla0tsTXRYRm1mVXRDS0FsSTNtZnBRSUNlNUJpcDdfZkdMYUNEMEUzX3JzdXVFMnF2Ull5QXIxTmZKOUdvRURHOE5ib2JHeUVTQ2lQTkMtRFBuUmoySDhLc0R1TG9HYUdMelZZdUNTWVRlbHlqYTU2UkZzOTNONmVPa1ZaWHpIUzU5Z3dZQ09WUWtsLS1BZkZQLUt2Ym5vSFB1VUJTLXI5TzFONWZIZmdKVmlDVFFTUTlDSXB3N0ZnMV8xczZDRWI1MmREZ1h1RkV0Tk1UZFhtNFhsWFpSdjNqanN0MWE0LWZzUnUwaVVOUXJLUUJkU2VTSVpBN1gzaFJDRlZlVlNlY0lQZVVZLTRJTUJpV21ONnA4S3JlWFE2VGh4aE9TWGphTHJGT28?oc=5","source":"Tom's Hardware","lang":"en","date":"2026-07-11"}],"stale":false,"snapshot_at":"2026-11-10T09:35:25Z","history_start":"2026-09-06"}