NVIDIA /TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
★ 15,051⑂ 2,924
DALI15,051
stars
+49+59+288
2,924
forks
–
watchers
–
prs
–
issues
–
releases
–
discussions
README
TensorRT LLM =========================== TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework that enables you to customize and extend the system. Architecture | Performance | Examples | Documentation | Roadmap --- Tech Blogs [10/14] KV Cache Compression in TensorRT LLM ✨ ➡️ link [09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K ✨ ➡️ link [09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM ✨ ➡️ link [08/20] Evaluating Agentic Serving with Trace Replay and Job-Level Metrics ✨ ➡️ link [07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM ✨ ➡️ link [07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM ✨ ➡️ link [05/15] Joint Optimization of Agent Applications and TensorRT-LLM ✨ ➡️ link [04/03] Tuning CUDA Graph Batch Sizes for Higher Output Throughput ✨ ➡️ link [04/03] DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 ✨ ➡️ link [03/16] Optimizing MoE Communication with One-Sided AlltoAll Over NVLink ✨ ➡️ link [03/04] Sparse Attention in TensorRT LLM ✨ ➡️ link [02/06] Accelerating Long-Context Inference with Skip Softmax Attention ✨ ➡️ link [01/09] Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs ✨ ➡️ link Previous Blogs [10/13] Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary) ✨ ➡️ link [09/26] Inference Time Compute Implementation in TensorRT LLM ✨ ➡️ link [09/19] Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly ✨ ➡️ link [08/29] ADP Balance Strategy ✨ ➡️ link [08/05] Running a High-Performance GPT-OSS-120B Inference Server with TensorRT LLM ✨ ➡️ link [08/01] Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization) ✨ ➡️ link [07/26] N-Gram Speculative Decoding in TensorRT LLM ✨ ➡️ link [06/19] Disaggregated Serving in TensorRT LLM ✨ ➡️ link [06/05] Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP) ✨ ➡️ link [05/30] Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers ✨ ➡️ link [05/23] DeepSeek R1 MTP Implementation and Optimization ✨ ➡️ link [05/16] Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs ✨ ➡️ link Latest News [04/03] 🎨 TensorRT LLM now supports diffusion models for visual generation ➡️ link Previous News [08/05] 🌟 TensorRT LLM delivers Day-0 support for OpenAI's latest open-weights models: GPT-OSS-120B ➡️ link and GPT-OSS-20B ➡️ link [07/15] 🌟 TensorRT LLM delivers Day-0 support for LG AI Research's