{"full_name":"vllm-project/vllm","owner":"vllm-project","name":"vllm","url":"https://github.com/vllm-project/vllm","description":"A high-throughput and memory-efficient inference and serving engine for LLMs","homepage":"https://vllm.ai","topics":[],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/136984999?v=4","created_at":"2023-02-09T11:23:20Z","pushed_at":"2026-11-10T09:22:46Z","metrics":{"stars":96666,"forks":24626,"watchers":604,"issues":19707,"prs":45860,"releases":109,"issues_open":null,"issues_closed":null,"prs_open":null,"prs_closed":null,"prs_merged":null,"discussions":489},"growth":{"day":{"stars":77,"forks":43,"base_day":"2026-11-09"}},"history":{"dates":["2026-11-09","2026-11-10"],"stars":[96589,96666],"forks":[24583,24626],"watchers":[604,604],"issues_open":[null,null],"prs_open":[null,null]},"boards":{"total":157},"same_owner":[{"full_name":"vllm-project/vllm-omni","name":"vllm-omni","boards":{}}],"issues":[{"number":50576,"title":"[Feature]: SM8x (Ampere A100/A800) support for DeepSeek-V4-Flash / DeepSeek-V4-Flash-0731 DSpark","comments":128,"created_at":"2026-07-31T14:42:04Z"},{"number":27433,"title":"[Feature]: Batch Invariant Feature and Performance Optimization","comments":105,"created_at":"2025-10-23T19:38:03Z"},{"number":4194,"title":"[RFC]: Multi-modality Support on vLLM","comments":98,"created_at":"2024-04-19T07:51:48Z"},{"number":42426,"title":"[Bug]: Kimi-K2.6 intermittently outputs only \"!!!!!!!!!!\" in reasoning field with content null","comments":93,"created_at":"2026-05-12T13:31:55Z"},{"number":32335,"title":"[Feature]: Extract KV-Cache update from all attention backends","comments":59,"created_at":"2026-01-14T14:24:56Z"},{"number":44280,"title":"[Roadmap] Rust Frontend Feature Parity","comments":57,"created_at":"2026-06-02T05:28:21Z"},{"number":19854,"title":"[RFC]: KV cache offloading","comments":57,"created_at":"2025-06-19T10:52:43Z"},{"number":19001,"title":"[Usage]: How to use DeepSeek-R1-0528-Qwen3-8B with function call","comments":56,"created_at":"2025-06-01T13:18:43Z"}],"discussions":[{"number":17638,"title":"Structured Generation with Reasoning Parser in offline mode.","comments":9,"category":"General","created_at":"2025-05-04T17:25:19Z"},{"number":17628,"title":"Running Llama4 quantized on 2xH100 80GB","comments":2,"category":"Q&A","created_at":"2025-05-04T08:23:59Z"},{"number":17544,"title":"......lib/python3.12/site-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb","comments":5,"category":"Q&A","created_at":"2025-05-01T17:35:27Z"},{"number":17512,"title":"vLLM cannot connect to existing Ray cluster","comments":6,"category":"General","created_at":"2025-05-01T00:55:10Z"},{"number":17472,"title":"Determining Overall Speed for One Long Prompt","comments":10,"category":"Q&A","created_at":"2025-04-30T13:06:55Z"},{"number":17304,"title":"Why gfx908 (mi100) was deleted from the arch list?","comments":0,"category":"General","created_at":"2025-04-28T11:52:02Z"},{"number":17293,"title":"Any known integration with n8n ?","comments":3,"category":"Q&A","created_at":"2025-04-28T08:43:31Z"},{"number":17231,"title":"Many 0 Day user questions -  What is this vllm thing useful","comments":3,"category":"Q&A","created_at":"2025-04-26T15:35:11Z"}],"latest_release":{"name":"v0.33.0","tag":"v0.33.0rc3","published_at":"2026-11-02T10:10:05Z","body":"# v0.33.0\r\n\r\n## Highlights\r\n\r\nThis release features 927 commits from 389 contributors (122 new)!\r\n\r\n* **New models**: Reflection Beam-1.0 (501B MoE, 23B active) with its DFlash2 drafter (#62074), Yandex AliceAI-Foundation-80B-A3B (#57963), Recursive Flint (40B MoE, 2.5B active) (#62938), Muse Spark Open (#61879), Step-5 Preview (#61727, #63147), NVIDIA Alpamayo 2 Super (#62957), Nemotron-Labs Dory1.0-4B (#60149), Nemotron Parse 3.5 (#61651), MiniCPM-OCR 1.0 (#63095), Domyn Edge (#62544) and the ColUltraVBERT visual document retriever (#62237).\r\n* **PyTorch 2.14**: vLLM is upgraded to PyTorch 2.14.0, with torchvision 0.29.0 and Triton 3.8.0 (#52183).\r\n* **CUDA 12.9 dropped**: release wheels and images are CUDA 13.0 only. The CUDA 12.9 wheels and the `-cu129` / `-cu129-ubuntu2404` images are no longer published (#62575); use the default CUDA 13.0 artifacts instead.\r\n* **libtorch ABI stable CUDA wheel**: the CUDA wheel is now ABI stable with libtorch.\r\n* **AWS EFA image**: a new `vllm/vllm-openai:v0.33.0-efa` image (x86_64 and aarch64) bundles the AWS EFA libraries and the EFA build of the Mooncake transfer engine, for KV transfer and multi-node serving on EFA-enabled EC2 instances (#58768).\r\n* **`vllm/vllm` image alias**: every release image is now also published under the shorter `vllm/vllm` name, e.g. `vllm/vllm:v0.33.0`, `vllm/vllm:v0.33.0-efa` and `vllm/vllm:v0.33.0-rocm`.\r\n* **Speculative decoding**: the new XPress refiner for block-diffusion drafters such as DFlash and DSpark (#54448); FLy, an opt-in lossy verification policy with 1.04-1.31x decode throughput on MI355X and B300 (#53987); DFlash/DSpark no longer reserve draft slots in the token budget, taking DeepSeek-V4.1-Flash + DSpark on GB300 from 1373 to 3390 output tok/s in the PR's benchmark (#59468); quantized draft `lm_head` (`draft_lm_head_quantization`, +18% decode at concurrency 1) (#59973); lookahead block hashes so EAGLE/MTP prefix-cache hits keep their last block, on by default (#61589); and MTP drafts in the Transformers modeling backend (#61270).\r\n* **Faster decode on NVIDIA**: opt-in latency-specialized decode on GB300 for Kimi-K3 (1.5-2x at TP8/TP16, #61867) and DeepSeek-V4.1-Flash (18.3% faster decode step at batch 1, #61868); DeepSeek-V4/V4.1 MXFP4 experts run as block FP8 through Humming on Hopper (+9-21% decode on 4x H200, #61875); FA4 sparse MLA decode on Blackwell (#55866); FlashInfer DCP prefill on trtllm-gen with 39-48% lower mean TTFT at 100k input on 8xB200 (#62029); and decode-only full CUDA graphs under MRV2 prefill context parallelism (2.6x over eager, #62377).\r\n* **AMD ROCm**: the default ROCm image and wheels move to ROCm 10.0, with ROCm 7.2 still published as `-rocm72` (#58761); MonoKernel decode via `--monokernel` for DeepSeek-V4.1-Flash on MI300X/MI325X (2.32x P50 interactivity at concurrency 1, #60731) and on every MI355X layer (#61082), and for MiniMax-M3-MXFP4 on MI355X (16.6% lower TPOT, #62712, #62994); the `ROCM_OCTAVE` 3-4 bit KV cache (4.6x the KV tokens of fp16 on RX 7900 XTX, #59774); and gfx1250 enablement for gpt-oss-120b, DeepSeek-R1, DeepSeek-V4-Flash and MiniMax-M3 (#58800).\r\n* **API and serving**: the OpenAI-compatible `/v1/decisions` endpoint, on by default for supported generation models (#60465, #61550); token embeddings with optional late chunking on `/v1/embeddings` (#61772, #63051, #60164); `output_mode=\"derender\"` on `/inference/v1/generate` (#60082); a large Rust frontend parity push, including no more HTTP 500 on long conversations (#62374) and unsupported flags failing at startup (#62431); concurrent worker startup with TP > 1 (#60028); and a watchdog that dumps stack traces from stuck engine and worker processes (#55700).\r\n* **Breaking changes**: Medusa and the MLP speculator removed (#61701); the legacy Ray compiled-graph executor removed (#61712, #62228); `--mamba-cache-mode` deprecated (#60043); requests sent while generation is paused now get HTTP 503 (#51488); the NIXL connector needs NIXL 1.5 or newer (#6"},"readme":true,"readme_zh":false,"news":[{"title":"vLLM v0.28.0","url":"https://github.com/vllm-project/vllm/releases/tag/v0.28.0","hn":"https://news.ycombinator.com/item?id=49492067","source":"Hacker News","lang":"en","date":"2026-08-29","points":108,"comments":43},{"title":"Ask HN: Are you training and running custom LLMs and how are you doing it?","url":"https://news.ycombinator.com/item?id=37121384","hn":"https://news.ycombinator.com/item?id=37121384","source":"Hacker News","lang":"en","date":"2023-08-14","points":15,"comments":2},{"title":"PyTorch-native TPU back end for vLLM","url":"https://github.com/vllm-project/vllm-torchtpu","hn":"https://news.ycombinator.com/item?id=50158960","source":"Hacker News","lang":"en","date":"2026-10-19","points":6,"comments":0},{"title":"vLLM introduces memory optimizations for long-context inference","url":"https://github.com/vllm-project/vllm/releases","hn":"https://news.ycombinator.com/item?id=47643924","source":"Hacker News","lang":"en","date":"2026-04-04","points":5,"comments":0},{"title":"vLLM IR: A Functional Intermediate Representation for vLLM","url":"https://github.com/vllm-project/vllm/issues/32358","hn":"https://news.ycombinator.com/item?id=47681076","source":"Hacker News","lang":"en","date":"2026-04-07","points":4,"comments":0},{"title":"vLLM 0.29.0: Model Runner V2 is now the default","url":"https://github.com/vllm-project/vllm/releases/tag/v0.29.0","hn":"https://news.ycombinator.com/item?id=49625693","source":"Hacker News","lang":"en","date":"2026-09-09","points":3,"comments":0},{"title":"Vllm: High-throughput and memory-efficient inference and serving engine for LLMs","url":"https://github.com/vllm-project/vllm","hn":"https://news.ycombinator.com/item?id=37453478","source":"Hacker News","lang":"en","date":"2023-09-10","points":3,"comments":0},{"title":"Vllm","url":"https://github.com/vllm-project/vllm","hn":"https://news.ycombinator.com/item?id=37022312","source":"Hacker News","lang":"en","date":"2023-08-06","points":3,"comments":0},{"title":"What did DeepSeek release? The scope of the engine as read from GitHub","url":"https://note.com/consuloji_ch/n/n29df1c22fa0b?hl=en","source":"note","lang":"en","date":"2026-11-05"},{"title":"Qualiteg Inc. on X: \"Sources: GitHub Releases https://t.co/Xgx4c3NVCI / PyPI https://t.co/JjjOY6lX9d\"","url":"https://x.com/Qualiteg_global/status/2117401093823152441","source":"x","lang":"en","date":"2026-11-02"},{"title":"DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash","url":"https://www.marktechpost.com/2026/10/20/deepseek-deepinfer-open-source-inference-framework/","source":"MarkTechPost","lang":"en","date":"2026-10-20"},{"title":"DeepSeek open-sources DeepInfer, the engine that runs its AI models","url":"https://thenextweb.com/news/deepseek-deepinfer-open-source-inference-engine","source":"The Next Web","lang":"en","date":"2026-10-20"},{"title":"[20261019] LLM News & FP8 KV Cache for Increasing Inference Headroom","url":"https://note.com/jazzy_briony8396/n/n7fe650352f0d?hl=en","source":"note","lang":"en","date":"2026-10-19"},{"title":"Intel LLM-Scaler-vLLM PV 1.5 Brings Big Improvements","url":"https://www.phoronix.com/news/Intel-LLM-Scaler-vLLM-PV-1.5","source":"Phoronix","lang":"en","date":"2026-10-15"},{"title":"Critical LMCache Flaw Lets Unauthenticated Attackers Run Code Remotely","url":"https://thehackernews.com/2026/10/unpatched-critical-lmcache-flaw-lets.html","source":"The Hacker News","lang":"en","date":"2026-10-07"},{"title":"Inferact launches with $150M in funding to commercialize vLLM","url":"https://siliconangle.com/2026/01/22/inferact-launches-150m-funding-commercialize-vllm/","source":"SiliconANGLE","lang":"en","date":"2026-01-22"},{"title":"Open Source Project With Little Revenue In Talks To Raise At Least $160 Million","url":"https://www.forbes.com/sites/iainmartin/2025/12/22/open-source-project-with-little-revenue-in-talks-to-raise-at-least-160-million/","source":"Forbes","lang":"en","date":"2025-12-22"},{"title":"蚂蚁集团和vLLM联合出了一个高性能量化推理Kernel 库：Humming地址：github.com/vllm-project/humming官方介绍：vllm.ai/blog/2026-10-21-humming这个库能让量化模型跑的更快。比如MXFP4量化后的DeepSeek-V4.1-Flash模型，测试速度能从442 tokens/s提升","url":"https://news.google.com/rss/articles/CBMiY0FVX3lxTFB2MDRoclpUTWlYLTQxR3B2MlRrYTVINmlvd0xvUmVnX1ZHSWFWRVMxdFJIU01kMlpkYU9JQmlQS2hLN1hKczd0djhBbEZiSkdMb21VQUhDMVhlbXB4ckZ3VHJrbw?oc=5","source":"新浪网","lang":"zh","date":"2026-10-22"},{"title":"Announcing New vLLM and SGLang Inference Engines on TPU","url":"https://news.google.com/rss/articles/CBMilAFBVV95cUxNRnUwYklFbjNPQ3haNEk3QmFCdlZHVUJpaVEzdXcxQWo3dkxLdjhXVnYxRFdjTmFtTnF1RzhIZ04wWUJQY3ZVTko2YnpRMW9TRXpGYXBYRHBEUWtXWHBzZ0dXSFM4WjFkSmdtaFlOZmh4aWRvdS02a2FtbzZvNzI1SjBkLXNqYk5rTG1QSUdjZV9fSUJu?oc=5","source":"Google for Developers Blog","lang":"en","date":"2026-10-20"},{"title":"千问、GLM、混元接连上新：FlagOS 四天三次 Day0，将前沿模型多芯适配变成可持续能力","url":"https://news.google.com/rss/articles/CBMiSEFVX3lxTE84Y2dJS1NxUkpqcHdQckh1cVQzSS05UEFZYVQ3c0NvTm5YWXY3V2haZ19qNGs3RmNmYXFQNlhESU5lb1ZxMmNiRg?oc=5","source":"智源社区","lang":"zh","date":"2026-09-01"},{"title":"The Open Source Stack Every AI Startup Uses: 7 GitHub Repos (With Real Star Counts)","url":"https://news.google.com/rss/articles/CBMimAFBVV95cUxNbUtrR05RRWJRTmZ1Q0dEeGdpNjREbTFIc3NkNkR0QVRfNFBZM1VWWnNIbEtVUnJ0ZE9vY0VkNXdSZTZJbEg4c3ZOZW1sc3lNdlROalF6UTlQUlVTamVwM3NiZm1vR1BVMkNiUkZDREVrbFhDRkpMc1dVd0JSMklGT1hQNXdCb2w2S3Yzb20yQW9pQzczVE9HUQ?oc=5","source":"Pasquale Pillitteri","lang":"en","date":"2026-08-13"},{"title":"每家AI初创公司都在用的开源技术栈：7个GitHub仓库（附真实star数）","url":"https://news.google.com/rss/articles/CBMikwFBVV95cUxQLVpBU1h3bks4TjF5R3l1cW1EZG1Ga3hFa2ZjVU1laWhCUWhsVFlKektkME1nNHJwYUhIVHNILURfaUpFYmJ6cnZvTEczTmpVTWxYR2M1NDRTRHdmUURScjdmTjBtT1FoTzJ0cDBoWTZwVmNnWURka1g0X1Nrb2NBNXhYUllMQnk2QWlrM3hwQjNfWlk?oc=5","source":"Pasquale Pillitteri","lang":"zh","date":"2026-08-13"},{"title":"vLLM-Omni Open Source: Integrating Diffusion Models, ViT, and LLM into a Pipeline, Completing Multimodal Inference in One Go","url":"https://news.google.com/rss/articles/CBMiSkFVX3lxTE1YS3RkamNGLWM0Zzg1TXZCLVpOMmlBSHZDM29KVmVZVXZzZk9wYkM3TEZVZlkyTnFRQlNGVXM3WE55OE5qbndJbkhB?oc=5","source":"AIBase","lang":"en","date":"2025-12-02"},{"title":"vLLM-Omni 上线：多模态推理更简单、更快、更省","url":"https://news.google.com/rss/articles/CBMiWEFVX3lxTE1pdXV1VVRxQk9sb2h0U09MQkVKSVBUcFFhR2NQX0NydW1YU3N4eGh1NUxwYmlqOUlDUDNIU24yQmVrTHZ3TGFrZjRFam00WjA3VHN5eVhaQmI?oc=5","source":"OSCHINA","lang":"zh","date":"2025-12-02"},{"title":"vLLM-Omni开源:把扩散模型、ViT、LLM塞进一条流水线，多模态推理一次跑完","url":"https://news.google.com/rss/articles/CBMiTkFVX3lxTFBTNzVUcWxwbDJyUnVka1BPRGg2LXhTNmVlZVZ3N3pJODNheEsxeU5NcTY4Sk5Rcm05VmhjV1BnQXY4OHA4TzZrRlBnOEpoZw?oc=5","source":"AIBase","lang":"zh","date":"2025-12-02"},{"title":"Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1","url":"https://news.google.com/rss/articles/CBMiwgFBVV95cUxQWHdidEFJV2lHZHBfMXN4YTAxclFWX2hZYlY5OFVwei12MWRGX25wQmlTNk1yWDkwUDhQNF92YUJ6NE5jb1dmZXRNNWlBQ0c1UWRpWEhtT3dZSmZ0OTdObjk3NDJia3dhUUwxYzRINlF3aGVDOUxvUUZkT0dKcThXa0hnUkNWM1hBT2xpcW1aSG9WdmRHSGpzTkZDZjJyMUt0Ull0TERSTnVIM29fcHFMNVRLTkZJbXZSUW44UEUyYlNpUQ?oc=5","source":"The GitHub Blog","lang":"en","date":"2025-10-28"},{"title":"Deploy LLMs on Amazon EKS using vLLM Deep Learning Containers","url":"https://news.google.com/rss/articles/CBMirAFBVV95cUxPMDJXRDg2akJrYVY1aDRiSklJeGY2Z09ReGlyVUhuX2g4RzFyVGNUeW83RWhyWks1UUpDYkltcm1lTzl1MkNlLW1JOGZGSkxWWXNyT2RGQnFNYzFKWEZTZHVVOFhhbUNzQ2k1YVRPVlN2QUF0QU83R2VHZmt5dm11Z1VZYjBqaHg4U0g5Q3NhbXRHMm9oTUV6RHJzbHduSE1iQkVJSS1aNUxuUDRj?oc=5","source":"Amazon Web Services (AWS)","lang":"en","date":"2025-08-14"}],"stale":false,"snapshot_at":"2026-11-10T09:35:25Z","history_start":"2026-09-06"}