README

DeepGEMM

DeepGEMM is a unified, high-performance tensor core kernel library that brings together the key computation primitives of modern large language models — GEMMs (FP8, FP4, BF16), fused MoE with overlapped communication (Mega MoE), MQA scoring for the lightning indexer, HyperConnection (HC), and more — into a single, cohesive CUDA codebase. All kernels are compiled at runtime through DeepJIT, requiring no CUDA compilation during installation.

DeepGEMM leverages some concepts from CUTLASS and CuTe, but avoids heavy reliance on their templates or algebras. The library is designed for simplicity, with only a limited number of core kernel functions, making it a clean and accessible resource for learning NVIDIA GPU kernel optimization techniques.

Despite its lightweight design, DeepGEMM's performance matches or exceeds expert-tuned libraries across various matrix shapes.

 News

- 2026.09.30
  - DeepGEMM Ascend is available! Check DeepGEMM-Ascend for more details
  - Add more optimizations, including locality domain features, check 462 for more details
- 2026.09.10: Sparse Indexer, Mega Gate, Mega mHC, DeepJIT, MoE and Indexer optimizations and more.
    - Please see 432 for more details.
- 2026.04.16: Mega MoE, FP8xFP4 GEMM, FP4 Indexer, PDL, faster JIT compilation and more.
    - Please see 304 for more details.
    - For Mega MoE benchmarks, refer to 316.
- 2025.09.28: DeepGEMM now supports scoring kernels (weighted ReLU MQA logits) for the lightning indexer for DeepSeek v3.2.
    - Please see 200 for more details.
- 2025.07.20: DeepGEMM now supports both SM90/SM100, and has a full refactor with a low-CPU-overhead JIT CPP module.
    - As NVCC 12.9 will automatically do the FFMA interleaving, all post optimizations will be no longer supported.
    - Please see 112 for more details.
- 2025.05.14: DeepGEMM now offers weight gradient kernels for dense and MoE backward! See 95 for details.
- 2025.04.18: DeepGEMM now achieves up to 1550 TFLOPS on H800! See 74, 78, 81, 86 and 340d988 for details.

 Quick start

 Requirements

- NVIDIA SM90 or SM100 architecture GPU
- Python 3.8 or higher
- Compilers and standard libraries with C++20  support
- CUDA Toolkit 12.9 or higher
- PyTorch 2.3 or higher
- CUTLASS 4.0 or higher (could be cloned by Git submodule)

 Development

bash
 Submodule must be cloned
git clone --recursive [email protected]:deepseek-ai/DeepGEMM.git
cd DeepGEMM

 Link some essential includes and build the C++ extension
cat develop.sh
./develop.sh


 Installation

bash
cat install.sh
./install.sh


Then, import deep_gemm in your Python project, and enjoy!

 Interfaces

 Notices

This library provides optimized GEMM kernels for NVIDIA GPUs with a naming convention: D = C + A @ B. The input shape layout is NT (non-transposed A, transposed B). While the SM90 implementation supports only the NT memory layout (row-major, col-major), the SM100 implementation supports all memory layouts (NT, TN, NN, TT). For example, fp8_gemm_nt will do a D = C