README

AnimateDiff

This repository is the official implementation of AnimateDiff [ICLR2024 Spotlight].
It is a plug-and-play module turning most community text-to-image models into animation generators, without the need of additional training.

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning 

Yuwei Guo,
Ceyuan Yang✝,
Anyi Rao,
Zhengyang Liang,
Yaohui Wang,
Yu Qiao,
Maneesh Agrawala,
Dahua Lin,
Bo Dai
(✝Corresponding Author)  





Note: The main branch is for Stable Diffusion V1.5; for Stable Diffusion XL, please refer sdxl-beta branch.


 Quick Demos
More results can be found in the Gallery.
Some of them are contributed by the community.


    
    
    
    
    
    

Model：ToonYou


    
    
    
    
    
    

Model：Realistic Vision V2.0


 Quick Start
Note: AnimateDiff is also offically supported by Diffusers.
Visit AnimateDiff Diffusers Tutorial for more details.
Following instructions is for working with this repository.

Note: For all scripts, checkpoint downloading will be automatically handled, so the script running may take longer time when first executed.

 1. Setup repository and environment


git clone https://github.com/guoyww/AnimateDiff.git
cd AnimateDiff

pip install -r requirements.txt


 2. Launch the sampling script!
The generated samples can be found in samples/ folder.

 2.1 Generate animations with comunity models

python -m scripts.animate --config configs/prompts/1_animate/1_1_animate_RealisticVision.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_2_animate_FilmVelvia.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_3_animate_ToonYou.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_4_animate_MajicMix.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_5_animate_RcnzCartoon.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_6_animate_Lyriel.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_7_animate_Tusun.yaml


 2.2 Generate animation with MotionLoRA control

python -m scripts.animate --config configs/prompts/2_motionlora/2_motionlora_RealisticVision.yaml


 2.3 More control with SparseCtrl RGB and sketch

python -m scripts.animate --config configs/prompts/3_sparsectrl/3_1_sparsectrl_i2v.yaml
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_2_sparsectrl_rgb_RealisticVision.yaml
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_3_sparsectrl_sketch_RealisticVision.yaml


 2.4 Gradio app
We created a Gradio demo to make AnimateDiff easier to use. 
By default, the demo will run at localhost:7860.

python -u app.py




 Technical Explanation

Technical Explanation

 AnimateDiff

AnimateDiff aims to learn transferable motion priors that can be applied to other variants of Stable Diffusion family.
To this end, we design the following training pipeline consisting of three stages.



- In 1. Alleviate Negative Effects stage, we train the domain ad