README

SGLang: Fast inference for LLMs and multimodal models






  
  
  



  Docs |
  Cookbook |
  Website |
  Blog |
  Slack


SGLang is an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion is its built-in image and video generation engine, included in this repository and the sglang Python package.

👋 Get started below, or meet the community at SGLang Events, including meetups, developer meetings, workshops, and office hours.

 SGLang Summit 2026: Meet the builders of open AI
 Our flagship annual gathering · November 12–13 · San Francisco

 Go deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.

 Get Started

Pull the Docker image, which includes SGLang and its dependencies:

bash
docker pull lmsysorg/sglang:latest


Alternatively, install SGLang in an activated Python environment with uv:

bash
uv pip install --prerelease=allow sglang


Next, launch your model:

- Quickstart: Run your first model and send a request.
- Cookbook: Choose your model and hardware to get a ready-to-run launch command.

 Supported Hardware

SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platforms.

| Platform | Representative hardware |
| --- | --- |
| NVIDIA | B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin |
| AMD | Instinct MI300X, MI325X, MI350X, MI355X |
| Google TPU | v6e, v7; SGLang-TorchTPU / SGL-JAX |
| Intel (GPU / CPU) | Arc / Arc Pro B-Series GPUs, Xeon CPUs |
| Apple Silicon | Macs via Metal / MLX |
| Huawei Ascend | A2, A3, 950PR/DT NPUs |
| Moore Threads | MTT S5000 GPUs |

Integrations in progress: AWS Trainium, Alibaba T-Head PPU, Cambricon MLU, Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.

SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.

See the Cookbook and platform guides for model compatibility and setup.

 SGL Ecosystem

| Area | Projects | Purpose |
| --- | --- | --- |
| Diffusion | SGLang Diffusion | Built into SGLang for image and video generation with diffusion models. |
| Audio | SGLang Omni | Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR). |
| Speculative Decoding | SpecForge | Train draft models for speculative decoding and deploy them with SGLang. |
| RL and Post-Training | Miles, slime, AReaL, Tunix, verl | Training frameworks that integrate SGLang for rollout generation. |
| Deployment and Orchestration | SMG, RBG, llm-d, Ray Serve, NVIDIA Dynamo | Deploy and scale SGLang inference services wi

Top issues

Discussions

Latest release

Highlights

1007 PRs from 257 contributors.

 New models

| Model | Type | Cookbook |
|---|---|---|
| Muse Spark Open | LLM / VLM | link |
| Beam-1.0 | LLM / VLM | link |
| MiniMax-M3.1-Flash | LLM / VLM | link |
| MiniCPM-V 4.7 | LLM / VLM | link |
| Kev | LLM / VLM | link |
| Cosmos3-Nano-Sim (Bimanual, Transfer, Auto) | Diffusion | link |
| FastH3 Omni | Diffusion | link |
| FLUX.3 Image Klein | Diffusion | link |
| HY-Nomas | Diffusion | link |
| MOSS-Image | Diffusion | link |

 What's new

 DeepSeek-V4.1 on GB300 gets up to 1.5x more prefill throughput under load and