Video Diffusion Models

By Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, David J. Fleet. 2022

2,743 citations recorded by the source corpus.

Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Diffusion-based generative models Recent advances in diffusion models [59] have enabled state-of-the-art image synthesis [10, 18, 19, 53, 55, 60] as well as generative models of other modalities such as video [21, 58], audio [30], text [35] and network parameters [45].”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Common diffusion models for images and videos can be roughly divided into pixel diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Dhariwal & Nichol, 2022; Ho et al., 2022a; Karras et al., 2022) and latent diffusion models (Rombach et al., 2022).”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “They are also rapidly finding use in other domains such as audio [27, 37] and video [19] generation, image segmentation [4, 54] and language translation [34].”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-image generation [7], speech synthesis [8, 9] and lossless compression [10].”
Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022, positive): “We call this approach the replacement method (following [16]) and make it explicit in Appendix Algorithm 2.”
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, neutral): “…new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and ﬁles in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history [52], agent workﬂow with long horizon [61]).”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, methodology): “…utilize the generative prior of the text-to-image model for our pose-to-video generation, following previous Video Diffusion Model baselines [10, 8, 13, 17], we inﬂate ﬁrst convolution layer of the pretrained U-Net to a 1 × 3 × 3 convolution kernels as a pseudo-3D convolutional layer for video…”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Inspired by these results, recent works have explored text-conditional generation in other modalities, such as video (Hong et al., 2022; Singer et al., 2022; Ho et al., 2022b;a) and 3D objects (Jain et al., 2021; Poole et al., 2022; Lin et al., 2022a; Sanghi et al., 2021; 2022).”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Recently, several text-to-video diffusion models were proposed and achieved high-quality video generation results [20, 25, 29, 67, 85].”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “…not available publicly, we replicated this method in two different ways: (1) using the UniPi model from HiP [2], which is a PVDM [62] latent video diffusion model, and (2) by implementing our own image-space video diffusion model with a factorized spatial-temporal 3D UNet similar to Ho et al. [27].”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Many studies[10, 16, 17, 23, 26, 30, 40, 52, 54] explore the augmentation of inter-frame attention modeling on the foundation of text-to-image (T2I) models to achieve video generation.”
Continuous diffusion for categorical data (2022, methodology): “Generative models have seen a rapid increase in scale and capabilities over the past few years, across many modalities, including images, audio signals, video and text (Borsos et al., 2022; Brown et al., 2020; Dhariwal et al., 2020; Ho et al., 2022a; Ramesh et al., 2022; Sa-haria et al., 2022b).”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “Recent general text-to-video generation approaches [7,12,33] pro-pose incorporating temporal modeling into the original T2I models and tuning the models on the video datasets.”
Machine Learning for Synthetic Data Generation: a Review (2023, background): “…SD2011 CycleGAN [30] vision GAN CNN pix2pix DP-CGAN [31] vision GAN deep CGAN MNIST BigGANs [32] vision GAN large scale GAN ImageNet VideoDiff [33] vision diffusion CNN BAIR Robot Pushing, Kinetics-600 VQ-VAE [34] vision VAE PixelCNN ImageNet GIRAFFE [35] vision GAN CNN CompCars, LSUN…”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based generation has been extended to other modalities including audio [35, 17, 28, 70], video [75, 91, 85, 25], and 4D fields [76].”
Flexible Diffusion Modeling of Long Videos (2022, methodology): “The combination of this diffusion process and a data distribution q ( x 0 , y ) (recalling that x 0 = x ) defines the joint distribution q ( x 0: T , y ) = q ( x 0 , y ) DDPMs work by “inverting” the diffusion process: given values of x t and y a neural network is used to parameterize p θ ( x t − 1 ”