MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

By Zhongcong Xu, Jianfeng Zhang, J. Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, Mike Zheng Shou. Computer Vision and Pattern Recognition, 2023

406 citations recorded by the source corpus.

This paper studies the human image animation task, which aims to generate a video of a certain reference iden-tity following a particular motion sequence. Existing an-imation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce Magic/snimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two inno-vations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available at https://showlab.github.io/magicanimate.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/9d587b3f1d0608bf89fdb4ea6eed3312a73e938c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zuozhuo Dai
2. Junming Chen
3. Yao Yao
4. Siyu Zhu
5. Xun Cao
6. Hao Zhu
7. Yinghui Xu
8. Shenhao Zhu

Citations in the corpus

• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “To provide enhanced guidance, some works adopt other signals, e.g. , depth/skeleton sequence, to precisely control the scene/human motion in the generated videos (Guo et al., 2023a; Chen et al., 2023c; Zhang et al., 2023c; Khachatryan et al., 2023; Hu et al., 2023; Xu et al., 2023).”
• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “This type of methods generates videos with other auxiliary controls, such as depth maps [27, 31], pose maps [89, 37, 83, 56], RGB images [5, 15, 61], or other guided motion videos [100, 86].”
• Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “MagicAnimate [47] follows a similar approach but employs a ControlNet tailored for DensePose [10] inputs instead of the more commonly used OpenPose [5] keypoints to provide more precise pose guidance.”