StreamDiffusion

 StreamDiffusion: A Pipeline-Level Solution for Real-Time Interactive Generation

Authors: Akio Kodaira*, Chenfeng Xu*, Toshiki Hazama*, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, Masayoshi Tomizuka, Kurt Keutzer

StreamDiffusion is an innovative diffusion pipeline designed for real-time interactive generation. It introduces significant performance enhancements to current diffusion-based image generation techniques.

We sincerely thank Taku Fujimoto and Radamés Ajna and Hugging Face team for their invaluable feedback, courteous support, and insightful discussions.

Key Features

1. Stream Batch

   • Streamlined data processing through efficient batch operations.

2. Residual Classifier-Free Guidance - Learn More

   • Improved guidance mechanism that minimizes computational redundancy.

3. Stochastic Similarity Filter - Learn More

   • Improves GPU utilization efficiency through advanced filtering techniques.

4. IO Queues

   • Efficiently manages input and output operations for smoother execution.

5. Pre-Computation for KV-Caches

   • Optimizes caching strategies for accelerated processing.

6. Model Acceleration Tools

   • Utilizes various tools for model optimization and performance boost.

When images are produced using our proposed StreamDiffusion pipeline in an environment with GPU: RTX 4090, CPU: Core i9-13900K, and OS: Ubuntu 22.04.3 LTS.

model	Denoising Step	fps on Txt2Img	fps on Img2Img
SD-turbo	1	106.16	93.897
LCM-LoRA + KohakuV2	4	38.023	37.133

Feel free to explore each feature by following the provided links to learn more about StreamDiffusion's capabilities. If you find it helpful, please consider citing our work:

@article{kodaira2023streamdiffusion,
      title={StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation},
      author={Akio Kodaira and Chenfeng Xu and Toshiki Hazama and Takanori Yoshimoto and Kohei Ohno and Shogo Mitsuhori and Soichi Sugano and Hanying Cho and Zhijian Liu and Kurt Keutzer},
      year={2023},
      eprint={2312.12491},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Installation

Step0: clone this repository

git clone https://github.com/cumulo-autumn/StreamDiffusion.git

Step1: Make Environment

You can install StreamDiffusion via pip, conda, or Docker(explanation below).

conda create -n streamdiffusion python=3.10
conda activate streamdiffusion

OR

python -m venv .venv
# Windows
.\.venv\Scripts\activate
# Linux
source .venv/bin/activate

Step2: Install PyTorch

Select the appropriate version for your system.

CUDA 11.8

pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu118

CUDA 12.1

pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu121

details: https://pytorch.org/

Step3: Install StreamDiffusion

For User

Install StreamDiffusion

#for Latest Version (recommended)
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]


#or


#for Stable Version
pip install streamdiffusion[tensorrt]

Install TensorRT extension

python -m streamdiffusion.tools.install-tensorrt

(Only for Windows) You may need to install pywin32 additionally, if you installed Stable Version(pip install streamdiffusion[tensorrt]).

pip install --force-reinstall pywin32

For Developer

python setup.py develop easy_install streamdiffusion[tensorrt]
python -m streamdiffusion.tools.install-tensorrt

Docker Installation (TensorRT Ready)

git clone https://github.com/cumulo-autumn/StreamDiffusion.git
cd StreamDiffusion
docker build -t stream-diffusion:latest -f Dockerfile .
docker run --gpus all -it -v $(pwd):/home/ubuntu/streamdiffusion stream-diffusion:latest

Quick Start

You can try StreamDiffusion in examples directory.

Real-Time Txt2Img Demo

There is an interactive txt2img demo in demo/realtime-txt2img directory!

Real-Time Img2Img Demo

There is a real time img2img demo with a live webcam feed or screen capture on a web browser in demo/realtime-img2img directory!

Usage Example

We provide a simple example of how to use StreamDiffusion. For more detailed examples, please refer to examples directory.

from  https://github.com/cumulo-autumn/StreamDiffusion