Stochastic Video Generation with a Learned Prior

By Emily L. Denton, Rob Fergus. 2018

587 citations recorded by the source corpus.

Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry generations, or both. In this paper we introduce an unsupervised video generation model that learns a prior model of uncertainty in a given environment. Video frames are generated by drawing samples from this prior and combining them with a deterministic estimate of the future frame. The approach is simple and easily trained end-to-end on a variety of datasets. Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “Work in this category includes variants of variational autoencoders (Babaeizadeh et al., 2018; Denton & Fergus, 2018).”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…been proposed for video prediction on the BAIR Robot dataset (Finn et al., 2016; Ebert et al., 2017; Babaeizadeh et al., 2017; Denton et al., 2017; Denton & Fergus, 2018; Lee et al., 2018) where the future frames are predicted given the past frame(s) and (or) action(s) of a robot arm moving…”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “…al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doerr et al., 2018; Gregor and Besse, 2018).”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “More recent work, based on VRNNs, uses temporal stochastic variables, e.g., the SV2P model of Babaeizadeh et al. (2018) and the SVGLP model of Denton & Fergus (2018).”
Learning to predict the cosmological structure formation (2018, background): “Deep learning (11) is a fast growing branch of machine learning where recent advances have lead to models that reach and sometimes exceed human performance across diverse areas, from analysis and synthesis of images (12–14), sound (15, 16), text (17, 18) and videos (19, 20) to complex control and Fi”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “This is addressed in stochastic latent variable models trained using variational inference to generate multiple plausible videos given a sequence of images (Babaeizadeh et al., 2017; Denton and Fergus, 2018).”
Stochastic Adversarial Video Prediction (2018, methodology): “Prior [1] and concurrent [2] work has explored stochastic models for video prediction, using the framework of variational autoencoders (VAEs) [5].”
Preventing Posterior Collapse with delta-VAEs (2019, methodology): “Our sequential image generation model resembles latent variable models used for timeseries (Chung et al., 2015; Babaeizadeh et al., 2017; Denton & Fergus, 2018) but does not rely on KL annealing, and has an additional autoregressive dependence of the outputs over time (rows of the image).”
A General Method for Amortizing Variational Filtering (2018, methodology): “For instance, many filtering inference methods re-use various components of the generative model [7, 12, 15, 9], while some methods introduce separate recurrent neural networks into the filtering procedure [4, 9] or encode the previous latent sample [26].”
Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods (2018, background): “CVAEs have been successfully applied on a large variety of tasks, include conditional image generation (Bao et al., 2017), next frame synthesis (Xue et al., 2016), video generation (Babaeizadeh et al., 2018; Denton & Fergus, 2018), trajectory prediction (Lee et al., 2017) among others.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “We use a common encoder-recurrent-decoder architecture (Denton and Fergus (2018); Hafner et al. (2018)).”
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, unknown): “Then, [7] proposed SVG.”
VideoFlow: A Flow-Based Generative Model for Video (2019, methodology): “Although methods based on variational autoencoders (Babaeizadeh et al., 2017; Denton & Fergus, 2018; Lee et al., 2018), and pixel-level autoregressive models (Hochreiter & Schmidhuber, 1997; Graves, 2013; van den Oord et al., 2016b;c; Van Den Oord et al., 2016) have previously been studied for…”
SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture (2020, background): “One reason for this is the use of MSE as guiding loss which is biased towards blurry images [32].”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, unknown): “SMMNIST ( 64 × 64 ) KTH ( 64 × 64 ) BAIR ( 64 × 64 ) p + f k n PSNR ↑ SSIM ↑ p + f k n PSNR ↑ SSIM ↑ p + f k n PSNR ↑ SSIM ↑ SVG-LP Denton and Fergus [2018] 18 7 100 13.543 0.741 18 7 100 28.131 0.883 18 7 100 18.648 0.846 FSTN Lu et al. [2017] 18 7 100 14.730 0.765 18 7 100 29.431 0.899 18 7 100 19”
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives (2018, background): “…such as natural images (Rezende & Mohamed, 2015; Kingma et al., 2016; Chen et al., 2016; Gulrajani et al., 2016), speech and music time-series (Chung et al., 2015; Fraccaro et al., 2016; Krishnan et al., 2015), and video (Babaeizadeh et al., 2017; Ha & Schmidhuber, 2018; Denton & Fergus, 2018).”
Diversity-Sensitive Conditional Generative Adversarial Networks (2019, methodology): “We download the pre-processed videos from Villegas et al. (2017); Denton & Fergus (2018), which contains the frames with reasonable motions.”