Diffusion Probabilistic Modeling for Video Generation

By Ruihan Yang, Prakhar Srivastava, Stephan Mandt. Entropy, 2022

331 citations recorded by the source corpus.

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against six baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality and probabilistic frame forecasting ability for all datasets.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/1641774b55a471a23eb31b722ee05c2e032fec7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

ModelScope Text-to-Video Technical Report (2023, neutral): “Recent works have utilized diffusion models [53, 37] to generate authentic videos [66, 14, 21, 59].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Additionally, the Residual Video Diffusion (RVD) model [268] utilizes an autoregressive, end-to-end optimized video diffusion model.”
Flexible Diffusion Modeling of Long Videos (2022, background): “Contributions (1) At the highest level, we claim to have concurrently developed one of the first denoising diffusion probabilistic model (DDPM)-based video generative models [15, 39].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Recently, driven by the tremendous success of applying the diffusion model to image synthesis, multiple works have proposed to explore diffusion models for conditional and unconditional video synthesis [57, 12, 70, 61, 4, 22, 19, 57, 65, 33, 28, 1, 59].”
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2022, neutral): “It has been quickly popularized in other domains, such as video [e.g., 24, 92, 21], music [51], audio [e.g., 33, 40, 60], and text [41, 88], and more tasks such as image editing [97, 50].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “In contrast to previous work on DMs for video generation [24,31,33,93, 104], we first pre-train our Video LDMs on images only (or use available pre-trained image LDMs), thereby allowing us to leverage large-scale image datasets.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Recently, several text-to-video diffusion models were proposed and achieved high-quality video generation results [20, 25, 29, 67, 85].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Recent research has demonstrated the potential of employing diffusion models for video synthesis [6, 18, 23, 28, 32, 68].”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “Probabilistic sequence modeling plays a crucial role in diverse machine learning applications including natural language processing [6, 47], video prediction [31, 69] and decision making [3, 22].”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, neutral): “Multiple works have conducted a survey on diffusion models covering all fields [6, 129].”
MagicVideo: Efficient Video Generation With Latent Diffusion Models (2022, methodology): “[51] propose a two-step framework for video generation—using a deterministic model to predict the next frame given observed ones, then utilizing a stochastic diffusion model to synthesize the residual for correcting the next frame prediction.”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, background): “…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] ﬂow based models [24], and ﬁnally adversarial based methods [48, 39, 43, 9, 40, 27].”
How Artificial Intelligence Is Shaping Medical Imaging Technology: A Survey of Innovations and Applications (2023, background): “They can also be applied to video data [66,67].”