VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation

By Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liangsheng Wang, Yujun Shen, Deli Zhao, Jinren Zhou, Tien-Ping Tan. Computer Vision and Pattern Recognition, 2023

432 citations recorded by the source corpus.

A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution. Despite its recent success in image synthesis, applying DPMs to video generation is still challenging due to high-dimensional data spaces. Previous methods usually adopt a standard diffusion process, where frames in the same video clip are destroyed with independent noises, ignoring the content redundancy and temporal correlation. This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. The denoising pipeline employs two jointly-learned networks to match the noise decomposition accordingly. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation. We further show that our decomposed formulation can benefit from pre-trained image diffusion models and well-support text-conditioned video creation.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/26c6090b7e7ba4513f82aa28d41360c60770c618

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “We compare with the recent T2V models VideoFusion (Luo et al, 2023) and Vide-oLDM (Blattmann et al, 2023b) in terms of the visual quality in Fig.”
DeepCache: Accelerating Diffusion Models for Free (2023, background): “These models have demonstrated remarkable efficacy across diverse applications, being employed for the generation of images [21, 62, 66], text [11, 30], audio [6, 46], and video [19, 38, 58].”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, background): “Recently, the technological revolution brought by diffusion models [14, 41] in image synthesis [35] has been impressively remarkable, and it has also led to significant advancements in video synthesis [3,27,39].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “To ensure a fair comparison, we used the ofﬁcial implementation of VideoCrafter and VideoFusion [74] to generate the same number of videos with the same text prompts.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Recent research has demonstrated the potential of employing diffusion models for video synthesis [6, 18, 23, 28, 32, 68].”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, neutral): “…video synthesis (Voleti et al., 2022; Harvey et al., 2022; Zhou et al., 2022; Wu et al., 2022b; Blattmann et al., 2023b; Khachatryan et al., 2023; H¨oppe et al., 2022; Voleti et al., 2022; Yang et al., 2022; Nikankin et al., 2022; Luo et al., 2023; An et al., 2023; Wang et al., 2023b).”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Recently, driven by the tremendous success of applying the diffusion model to image synthesis, multiple works have proposed to explore diffusion models for conditional and unconditional video synthesis [57, 12, 69, 61, 4, 22, 19, 57, 64, 33, 28, 1, 59].”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, background): “Since the setting in this report is newly proposed, there is currently little work targeting it.”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “LaVie [104] 0.00% 6.51% 49.72% ModelScope [72, 98] 0.00% 33.72% 66.39% VideoCrafter [35] 0.00% 51.63% 89.72% CogVideo [41] 0.00% 14.19% 42.22% Temporal Flickering.”