AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

By Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai. International Conference on Learning Representations, 2023

1,619 citations recorded by the source corpus.

With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Signal Processing

View Paper: https://www.semanticscholar.org/paper/c1caa303549764d220ff17dc1785985dd1ba6047

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Signal Processing

Citations in the corpus

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “Simultaneously, diffusion-based methodologies [11, 18, 18, 43, 52] incorporate the reference image and various dynamics as conditions at both the appearance and motion levels.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Our approach contrasts with others, such as diffusion models, which typically solve these tasks by adopting multiple individually tuned adapter models to control the diffusion process [23, 28, 39].”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, background): “We choose openly-accessible top-performing text-to-video models as baselines, including T2V-Turbo (Li et al., 2024), AnimateDiff (Guo et al., 2023), VideoCrafter2 (Chen et al., 2024a), OpenSora (Zheng et al., 2024b), Show-1 (Zhang et al., 2023a), Gen-2 (runway, 2023), Pika (pik, 2023), and LaVie-2…”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, positive): “Early works propose inputting numerical camera parameters into their video generation models as an additional condition to fine-tune for camera control [15, 31, 47, 53, 59, 60].”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “AnimateDiff (Guo et al., 2023b) utilizes a pluggable motion module to enable high-quality animation creation on personalized image backbones (Ruiz et al., 2023).”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, neutral): “Currently, several open-source T2V models exist, i.e., ModelScope [50], Hotshot-XL [4], AnimateDiff [23], and Zeroscope V2 XL [7].”
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, positive): “Following prior works [10], we add sinusoidal positional encoding to make the model aware of the position for each frame within the video.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, neutral): “…Open-Sora-Plan(T2V) (Lab & etc., 2024), Lavie (Wang et al., 2023c), ModelScope (Wang et al., 2023b), OpenSora (Zheng et al., 2024), AnimateDiff (Guo et al., 2023), Open-Sora-Plan(TI2V) (Lab & etc., 2024), Dynamicrafter (Xing et al., 2023), EasyAnimate (Xu et al., 2024) through both Explicit…”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “AnimateDiff [53] and SVD [17] investigate class-conditioned video generation, grouping camera movements and utilizing LoRA [54] modules to create specific camera motions.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “For example, AnimateDiff [51] LoRA fine-tuned a video diffusion model with fixed types of camera motions, and MotionCtrl [52] conditioned the model on arbitrary specified camera trajectories.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Swayed by the success of diffusion models [24, 48, 100], a surge of diffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121].”
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models (2023, neutral): “–40], originally proposed by [1, 2].”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “We compare our approach with several state-of-the-art T2V models, including popular commercial models such as Gen-2 [5] and Pika Labs [9], as well as open-source models like Show-1 [60], VideoCrafter1 [15], and AnimateD-iff [22].”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “Due to the success of diffusion models in the field of image generation [72, 34], the exploration in the domain of video generation [28, 40, 75, 84, 91, 96, 58, 12, 90, 57] is also becoming popular.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “These advanced technologies enable educators to transform text descriptions or curriculum outlines into dynamic, engaging video content tailored to the specific style, and interests of individual learners [147, 148, 149, 150].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “We compare the zero-shot text-to-video generation of AnimateDiff [26] with the ofﬁcial weight (pretrained on 2.5 M videos) and our Panda-2M pretraining.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “Finally, we demonstrate that our model allows for explicit motion control by specifically prompting the temporal layers with motion cues and also via training LoRA-modules [32, 45] on datasets resembling specific motions only, which can be efficiently plugged into the model.”