Stochastic Adversarial Video Prediction

By Alex X. Lee, Richard Zhang, Frederik D. Ebert, Pieter Abbeel, Chelsea Finn, Sergey Levine. 2018

479 citations recorded by the source corpus.

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “However, these metrics were developed for images and have serious flaws when applied to videos [41, 40, 45, 23].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “In addition, action-conditioned samples are visually consistent with the input 3 SV2P (Babaeizadeh et al., 2017), SAVP (Lee et al., 2018), DVD-GAN-FP (Clark et al., 2019), Video Transformer (Weis-senborn et al., 2019), Latent Video Transformer (LVT) (Rakhimov et al., 2020), and TrIVD-GAN (Luc et…”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “There are also various GAN-based approaches, such as the SAVP approach of Lee et al. (2018) and the MoCoGAN approach of Tulyakov et al. (2018).”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “The video prediction architecture used in visual foresight is a deterministic variant of the SAVP video prediction model [47] based heavily on prior work [5].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “The SNA variant that we use incorporate the architectural improvements proposed by [49].”
Video-to-Video Synthesis (2018, methodology): “Conditioning on the observed frames, video prediction models are trained to predict future frames [15,18,36,41,42,45,50,65,68,71,72,77].”
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning (2018, methodology): “In this paper we use the video prediction model architecture developed by [35], where future images are generated by transforming past images.”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “Improving these models is an active research question [2, 19].”
Video Diffusion Models (2022, methodology): “Prior work on video generation has usually employed other types of generative models (e.g. Babaeizadeh et al., 2017; 2021; Lee et al., 2018; Kumar et al., 2019; Clark et al., 2019; Weissenborn et al., 2019; Yan et al., 2021; Walker et al., 2021).”
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning (2018, methodology): “…models as comparisons: • No physics is an ablation of our model that does not run the learned physics engine, but instead simply sets ¯O = O • Stochastic adversarial video prediction ( SAVP ), a block-box video prediction model which does not employ an object factorization Lee et al. (2018).”
Collaborative Learning for Faster StyleGAN Embedding (2020, background): “However, recently several works [26, 11, 22, 38] argue that PSNR and SSIM are not convincing enough to measure the similarity between the input image and the generated image.”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, methodology): “Indeed, pixel-level future predictors are standard in robotics applications [34, 68] due to the longstanding difficulty of inferring accurate object-centric representations from raw video data without supervision, despite recent progress [14, 64, 12].”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “Early research uses either regression loss [19, 48] or GAN loss [37, 57, 78, 81] to supervise the frame prediction task.”
Stochastic Image-to-Video Synthesis using cINNs (2021, methodology): “The assessment of diversity is typically based on measures utilizing feature representations of pretrained models [17, 34].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, unknown): “Prediction is a central problem in computer vision which researchers have been grappling with since the early days of the ﬁeld [10, 12, 22, 30, 35, 40, 56].”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, unknown): “inference with 2D recurrence [35], [51], [52], [53], [54], [55], [56].”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, methodology): “We observe that VideoFlow outperforms diversity values reported in prior work (Lee et al., 2018) while being competitive in the realism axis.”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “…where the decoder only outputs the mean parameter (Castrejon et al., 2019; Denton & Fergus, 2018; Lee et al., 2019; Babaeizadeh et al., 2018; Lee et al., 2018; Hafner et al., 2019b; Pong et al., 2019; Zhu et al., 2017; Pavlakos et al., 2019), or uses other ad-hoc modiﬁcations to the…”
Adversarial Video Generation on Complex Datasets (2019, methodology): “…Gated Recurrent Unit The dual discriminator D is updated twice for every update of G (Heusel et al., 2017) and we use Spectral Normalization (Zhang et al., 2018) for all weight layers (approximated by the ﬁrst singular value) and orthogonal initialization of weights (Saxe et al., 2013).”