Stochastic Variational Video Prediction

By Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, Sergey Levine. 2017

585 citations recorded by the source corpus.

Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images requires the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world video. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “These results are not strictly comparable, since prior work has used longer priming sequences of two (Babaeizadeh et al., 2018; Lee et al., 2018) or three (Kumar et al., 2019) frames, whereas our models (to our disadvantage) see a single prime frame.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “, the SV2P model of [4] and the SVGLP model of [5].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “…models with latent variables following the inﬂuential work by Kingma and Welling [37], Rezende et al. [59], and models optimized with this bound have been successfully used to model data such as natural images [58, 38, 11, 26], speech and music time-series [12, 23, 39], and video [2, 28, 17].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “Plenty of methods have been proposed for video prediction on the BAIR Robot dataset (Finn et al., 2016; Ebert et al., 2017; Babaeizadeh et al., 2017; Denton et al., 2017; Denton & Fergus, 2018; Lee et al., 2018) where the future frames are predicted given the past frame(s) and (or) action(s) of a…”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Inspired by Babaeizadeh et al. (2017a), we tried a variational autoencoder (Kingma & Welling, 2014) to model the stochasticity of the environment.”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “since we are ultimately interested in using the model for planning, but one may also use the full smoothing posterior during training (Babaeizadeh et al., 2017).”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “Independently from our work, Babaeizadeh et al. (2018) develop a stochastic sequence model and illustrate its representational power compared to deterministic models, using an example similar to the one we present in the Appendix.”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “More recent work, based on VRNNs, uses temporal stochastic variables, e.g., the SV2P model of Babaeizadeh et al. (2018) and the SVGLP model of Denton & Fergus (2018).”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Prior work on action-conditioned video prediction has studied predicting synthetic video game images [16], [29], 3D point clouds [19], and real-world images [30], [31], [32], using both direct autoregressive frame prediction [31], [32], [33] and latent variable models [34], [35].”
Stochastic Video Generation with a Learned Prior (2018, methodology): “First, the generated frames are significantly sharper with both our models (see direct comparisons to Babaeizadeh et al. (2017) in Figure Fig. 9). Second, training our model is much easier. Despite the same prior distribution being used for both our fixed-prior model and Babaeizadeh et al. (2017), t”
Stochastic Adversarial Video Prediction (2018, methodology): “Prior [1] and concurrent [2] work has explored stochastic models for video prediction, using the framework of variational autoencoders (VAEs) [5].”
Preventing Posterior Collapse with delta-VAEs (2019, methodology): “Our sequential image generation model resembles latent variable models used for timeseries (Chung et al., 2015; Babaeizadeh et al., 2017; Denton & Fergus, 2018) but does not rely on KL annealing, and has an additional autoregressive dependence of the outputs over time (rows of the image).”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “Improving these models is an active research question [2, 19].”
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning (2018, methodology): “Comparing with representative video prediction (Lee et al., 2018; Babaeizadeh et al., 2018) and physical prediction (Ehrhardt et al., 2017; Mottaghi et al., 2016; Li et al., 2017; Lerer et al., 2016) methods, our approach achieves substantially better results at tasks that require building…”
Deep Probabilistic Video Compression (2018, methodology): “BAIR action-free robot pushing dataset (Ebert et al., 2017) consists of a robot pushing objects on a table, which is also used in (Babaeizadeh et al., 2018; Denton & Fergus, 2018; Lee et al., 2018).”
Predicting Video with VQVAE (2021, background): “While some work in the area of video generation [7, 54, 45] has explored video synthesis—generating videos with no prior frame information—many approaches actually focus on the task of video prediction conditioned on past frames [41, 47, 38, 30, 23, 2, 34, 58, 60, 13, 27].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “Early work considers generating or warping pixels or optical flow to synthesize immediate futures [3, 11, 32, 41, 42, 46, 54, 55, 56, 57, 64].”
Hierarchical Relational Inference (2020, methodology): “using a variational approach (Babaeizadeh et al. 2018; Denton and Fergus 2018; Kumar et al. 2020), or (additionally) require adversarial training (Lee et al.”