MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

By Vikram S. Voleti, Alexia Jolicoeur-Martineau, C. Pal. arXiv.org, 2022

422 citations recorded by the source corpus.

Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -- when only future/past frames are masked; unconditional generation -- when both past and future frames are masked; and interpolation -- when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using $\le$ 4 GPUs. Project page: https://mask-cond-video-diffusion.github.io ; Code : https://github.com/voletiv/mcvd-pytorch

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/3890d82362d07064687a4b5e9024fc4c92921998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, background): “…42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] ﬂow based models [24], and ﬁnally adversarial based methods [48, 39, 43, 9, 40, 27].”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “Inspired by the notable success of the diffusion model in image synthesis, several recent studies have ventured into applying diffusion models for both conditional and unconditional video synthesis (Voleti et al., 2022; Harvey et al., 2022; Zhou et al., 2022; Wu et al., 2022b; Blattmann et al., 2023”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Recently, driven by the tremendous success of applying the diffusion model to image synthesis, multiple works have proposed to explore diffusion models for conditional and unconditional video synthesis [57, 12, 69, 61, 4, 22, 19, 57, 64, 33, 28, 1, 59].”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “These include text-to-video [56, 65, 81], image-to-video [78], video-to-video stylization [11, 15, 66], and video editing [10, 26, 68] among other video applications.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, background): “Video prediction can be regarded as a special form of generation, leveraging past observations to anticipate future frames [1, 9, 22, 26, 41, 59, 60, 65].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “Our training approach for prediction and interpolation models is inspired by recent works [24,33,93] that use similar masking techniques (also see Appendix C).”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “However, directly extending DMs to video synthesis requires substantial computational resources [12, 39].”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “Diffusion models implement an iterative refinement process by learning to gradually denoise a sample from a normal distribution and have been successfully applied to high-resolution text-to-image [13, 64, 68, 71, 75] and video synthesis [9, 29, 41, 82, 95].”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, background): “Diffusion models [14, 47] have recently shown to be powerful generative models that can generate a wide variety of images [3, 39, 40] and videos [2, 12, 51] by iteratively denoising a noise sample.”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “Generative adversarial networks [17] and variational auto-encoders [18] are the commonly used backbones in early research of video generation, e.g. , VGAN [47], TGAN [40], MoCoGAN [44], GODIA [52], StyleGAN-V [41], and MCVD [46].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Swayed by the success of diffusion models [24, 48, 100], a surge of diffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Two concurrent preprints also study a video diffusion model [72,73].”