README

📚 LeetCUDA: An open-source book of modern CUDA learning notes
  
      
       
      
      
      
      
      
  


[leetcuda-pdf]: https://github.com/xlite-dev/LeetCUDA/releases/download/v4.1.8/leetcuda-20261108.pdf
[leetcuda-readthedocs]: https://leetcuda.readthedocs.io/en/latest/

It covers Tensor/CUDA Cores, [200+ CUDA kernels][leetcuda-pdf], [HGEMM][leetcuda-pdf] achieving 98%~100% of cuBLAS TFLOPS, [FP8 & FP4 GEMM][leetcuda-pdf], and [Flash-Attention][leetcuda-pdf] built on Tensor Cores with pure MMA PTX. The LeetCUDA's open-source book is now available at [©️LeetCUDA.pdf][leetcuda-pdf] (600+ pages) and online docs at [leetcuda.io (zh, en)][leetcuda-readthedocs]. 🔥🔥🔥


  
      
  


Please also check out our production-ready Kernel Library: ffpa-attn - Fast and Memory-Efficient Exact Attention (BF16/FP16/FP8/FP4) for Large Headdim, 1.5x~15x🔥🔥 speedup over standard PyTorch SDPA.







 📖 Quick Start

bash
git clone https://github.com/xlite-dev/LeetCUDA.git && cd LeetCUDA
git submodule update --init --recursive --force && cd kernels/interview
 Install the latest cuBLAS & cuDNN for bench (remove the old version)
apt remove -y libcublas-cuda-13 libcudnn9-cuda-13 libcudnn9-dev-cuda-13
apt install -y cublas-cuda-13 cudnn9-cuda-13 && apt install -y ccache

bash
 Blackwell (e.g., RTX 5090, PRO 5000/6000, recommended: CUDA=13.2)
./build.sh --arch sm_120a && ./bin/leetcuda_bench_sm120a.bin --bench


 🤖 Agentic workflow

LeetCUDA provides a leetcuda-cpp-kernel SKILL that reuse the knowledge and examples from the open-source book and repository. Users can use it with Coding Agents, e.g, GitHub Copilot, Claude Code, Open Code.

bash
 Install the skill for different coding agents (recommended: gh=2.101.0)
gh skill install xlite-dev/LeetCUDA leetcuda-cpp-kernel --agent opencode
gh skill install xlite-dev/LeetCUDA leetcuda-cpp-kernel --agent claude-code
gh skill install xlite-dev/LeetCUDA leetcuda-cpp-kernel --agent github-copilot


 ©️Citations

BibTeX
@misc{LeetCUDA@2025,
  title={LeetCUDA: An Open-Source Book of Modern CUDA Learning Notes for Beginners},
  url={https://github.com/xlite-dev/LeetCUDA.git},
  note={Open-source software available at https://github.com/xlite-dev/LeetCUDA.git},
  author={DefTruth and Many Others},
  year={2025}
}



 📖 Click here to show the legacy contents 

 📖 Contents


- 📖 HGEMM-MMA 🎉🎉
- 📖 FlashAttention-MMA 🎉🎉
  - 📚 Split KV (Basic, FA-1)
  - 📚 Split Q (Faster, FA-2)
  - 📚 Split Q + Shared KV
  - 📚 Split Q + Shared QKV
  - 📚 Split Q + QK Tiling
  - 📚 Split Q + QKV Tiling
- 📖 200+ CUDA Kernels 🔥🔥
  - 📚 Easy ⭐️
  - 📚 Medium ⭐️⭐️
  - 📚 Hard ⭐️⭐️⭐️
  - 📚 Hard+ ⭐️⭐️⭐️⭐️
  - 📚 Hard++ ⭐⭐⭐️⭐️⭐️
  - 📚 Triton ⭐⭐⭐️
  - 📚 CUTLASS ⭐⭐⭐️
- 📖 100+ LLM/CUDA Blogs 🔥
- 📖 How to Contribute 👀👇


 📖 HGEMM Benchmark 🎉🎉



Currently, on NVIDIA L20, RTX 4090 and RTX 3080 Laptop, compared with cuBLAS's default Tensor Cores algorithm, the HGEMM (WMMA/MMA/CuTe) in this repo (blue🔵) can achieve 98%~100% of its (orange🟠) performance. Please che