README

StreamDiffusion

English | 日本語 | 한국어


  
  


 StreamDiffusion: A Pipeline-Level Solution for Real-Time Interactive Generation

Authors: Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, Masayoshi Tomizuka, Kurt Keutzer

StreamDiffusion is an innovative diffusion pipeline designed for real-time interactive generation. It introduces significant performance enhancements to current diffusion-based image generation techniques.




We sincerely thank Taku Fujimoto and Radamés Ajna and Hugging Face team for their invaluable feedback, courteous support, and insightful discussions.

 Key Features

1. Stream Batch

   - Streamlined data processing through efficient batch operations.

2. Residual Classifier-Free Guidance - Learn More

   - Improved guidance mechanism that minimizes computational redundancy.

3. Stochastic Similarity Filter - Learn More

   - Improves GPU utilization efficiency through advanced filtering techniques.

4. IO Queues

   - Efficiently manages input and output operations for smoother execution.

5. Pre-Computation for KV-Caches

   - Optimizes caching strategies for accelerated processing.

6. Model Acceleration Tools
   - Utilizes various tools for model optimization and performance boost.

When images are produced using our proposed StreamDiffusion pipeline in an environment with GPU: RTX 4090, CPU: Core i9-13900K, and OS: Ubuntu 22.04.3 LTS.

|            model            | Denoising Step | fps on Txt2Img | fps on Img2Img |
| :-------------------------: | :------------: | :------------: | :------------: |
|          SD-turbo           |       1        |     106.16     |     93.897     |
| LCM-LoRA + KohakuV2 |       4        |     38.023     |     37.133     |

Feel free to explore each feature by following the provided links to learn more about StreamDiffusion's capabilities. If you find it helpful, please consider citing our work:

bash
@article{kodaira2023streamdiffusion,
      title={StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation},
      author={Akio Kodaira and Chenfeng Xu and Toshiki Hazama and Takanori Yoshimoto and Kohei Ohno and Shogo Mitsuhori and Soichi Sugano and Hanying Cho and Zhijian Liu and Kurt Keutzer},
      year={2023},
      eprint={2312.12491},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}


 Installation

 Step0: clone this repository

bash
git clone https://github.com/cumulo-autumn/StreamDiffusion.git


 Step1: Make Environment

You can install StreamDiffusion via pip, conda, or Docker(explanation below).

bash
conda create -n streamdiffusion python=3.10
conda activate streamdiffusion


OR

cmd
python -m venv .venv
 Windows
.\.venv\Scripts\activate
 Linux
source .venv/bin/activate


 Step2: Install PyTorch

Select the appropriate version for your system.

CUDA 11.8

bash
pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu118