Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

By Liucheng Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, Liefeng Bo. Computer Vision and Pattern Recognition, 2023

832 citations recorded by the source corpus.

Character Animation aims to generating character videos from still images through driving signals. Currently, diffusion models have become the mainstream in visual generation research, owing to their robust generative capabilities. However, challenges persist in the realm of image-to-video, especially in character animation, where temporally maintaining consistency with detailed information from character remains a formidable problem. In this paper, we leverage the power of diffusion models and propose a novel framework tailored for character animation. To preserve consistency of intricate appearance features from reference image, we design ReferenceNet to merge detail features via spatial attention. To ensure controllability and continuity, we introduce an efficient pose guider to direct character's movements and employ an effective temporal modeling approach to ensure smooth inter-frame transitions between video frames. By expanding the training data, our approach can animate arbitrary characters, yielding superior results in character animation compared to other image-to-video methods. Furthermore, we evaluate our method on image animation benchmarks, achieving state-of-the-art results.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/c8dc4af5c61f95cc79b7f83e8339efa62af8f811

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Yinghao Xu
2. Bo Dai
3. Gordon Wetzstein
4. Ceyuan Yang

In Computer Vision and Pattern Recognition

1. Yinghao Xu
2. Ceyuan Yang
3. Bo Dai
4. Zuozhuo Dai
5. Gordon Wetzstein
6. Junming Chen
7. Zhiling Yan
8. Yuwei Guo

In Control and Systems Engineering

1. Bo Dai
2. Lifang He

Citations in the corpus

• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “This type of methods generates videos with other auxiliary controls, such as depth maps [27, 31], pose maps [89, 37, 83, 56], RGB images [5, 15, 61], or other guided motion videos [100, 86].”
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “We use the LAVIS (Li et al., 2023) to generate the text prompts for each video clip of the used dataset (Objaverse, MVImageNet, RealEstate10K, and ACID).”
• Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “In light of this, we are inspired by the advanced work [16] and propose a guidance encoder designed to encode our multilevel guidance.”
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Moreover, image-to-video editing techniques [151, 152, 153] present innovative avenues for converting static educational assets into interactive videos, thereby supporting a range of learning preferences and potentially increasing student engagement.”