DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

By Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Gongye Liu, Xintao Wang, Tien-Tsin Wong, Ying Shan. European Conference on Computer Vision, 2023

507 citations recorded by the source corpus.

Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that our proposed method can produce visually convincing and more logical&natural motions, as well as higher conformity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/083bab4a967c2221d9f4da9110fe37d8ca679078

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Ying Shan
2. Deli Zhao
3. Jingren Zhou
4. Xiang Wang
5. Hangjie Yuan
6. Shiwei Zhang
7. Yingya Zhang

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Deli Zhao
3. Jingren Zhou
4. Jinbo Xing
5. Jiwen Yu
6. Xiang Wang
7. Hangjie Yuan
8. Kang Zhao

In Control and Systems Engineering

1. Ying Shan

Citations in the corpus

• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “To analyze the generalization of Vista across different datasets, we compare it against three prominent general-purpose video generators trained on web-scale data [5, 133, 144] (see Fig.”
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “Similar strategies have also been employed in video generation, enabling controls like RGB images [17], [18], [47], depth [48], [49], trajectory [50], [51], and semantic maps [52].”
• WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “…(Wang et al., 2023b), OpenSora (Zheng et al., 2024), AnimateDiff (Guo et al., 2023), Open-Sora-Plan(TI2V) (Lab & etc., 2024), Dynamicrafter (Xing et al., 2023), EasyAnimate (Xu et al., 2024) through both Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, across three…”
• I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “As a concurrent method, Videocrafter1 [4, 50] leverages a diffusion model to address high-quality video generation tasks and places significant emphasis on fostering the growth of the community.”