An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders

By Jacob Walker, Carl Doersch, Abhinav Kumar Gupta, Martial Hebert. 2016

526 citations recorded by the source corpus.

In a given scene, humans can easily predict a set of immediate future events that might happen. However, pixel-level anticipation in computer vision is difficult because machine learning struggles with the ambiguity in predicting the future. In this paper, we focus on predicting the dense trajectory of pixels in a scene—what will move in the scene, where it will travel, and how it will deform over the course of one second. We propose a conditional variational autoencoder as a solution to this problem. In this framework, direct inference from the image shapes the distribution of possible trajectories while latent variables encode information that is not available in the image. We show that our method predicts events in a variety of scenes and can produce multiple different predictions for an ambiguous future. We also find that our method learns a representation that is applicable to semantic vision tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “Some use a single static stochastic variable, which is then “decoded” into a sequence using an RNN, either using a VAE-style loss [32, 35] or a GAN-style loss [31, 17].”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “For videos, self-supervision strategies include: leveraging temporal continuity via tracking [44, 45], predicting future [42], or preserving the equivariance of egomotion [13, 50, 30].”
Multi-task Self-Supervised Visual Learning (2017, background): “One can simply predict future [39, 40], although such predictions may be difﬁcult to evaluate.”
Generating Videos with Scene Dynamics (2016, background): “learn representations of video using unlabeled data. Our work is also related to efforts to predict the future in video [33, 22, 44, 51, 42, 17, 8, 55] as well as concurrent work in future generation [6, 15, 20, 50, 56, 43]. Often these works may be viewed as a generative model conditioned on the pa”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[112] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Some modeling choices include using pixels [28, 100], graphs [10], euclidean distance using sensors [2], or studying cycle consistency across time Epstein et al.”
Tutorial on Variational Autoencoders (2016, background): “While a CVAE works quite well for this purpose, unfortunately a regressor actually works quite well also, producing relatively crisp samples.”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “Some use a single static stochastic variable, as in a conditional VAE (Kingma & Welling, 2014), which is then “decoded” into a sequence using an RNN, either using a VAE-style loss (Walker et al., 2016; Xue et al., 2016) or a GAN-style loss (Vondrick et al., 2016b; Mathieu et al., 2016).”
Deep visual foresight for planning robot motion (2016, methodology): “Video prediction is an active area of research in deep learning and computer vision [36], [41], [42] and, with the method described in this work, improvements to such predictive models could translate into improvements in robotic manipulation capabilities.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “This approach differs from recent work on optic flow prediction [25], which predicts where pixels will move to using direct optical flow supervision.”
Toward Multimodal Image-to-Image Translation (2017, background): “While conditional VAEs [42] and autoregressive models [32, 33] have shown promise [16, 44, 46], imageto-image conditional GANs have lead to a substantial boost in the quality of the results.”
Stochastic Adversarial Video Prediction (2018, methodology): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Video-to-Video Synthesis (2018, background): “The video synthesis problem exists in various forms, including future video prediction [15,18,42,45, 50,65,68,71,77] and unconditional video synthesis [59,67,69].”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “VAEs have been used for image and video generation [7, 13, 28, 29, 33, 41, 42, 44].”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “This has obvious applications in tasks like future prediction [41], intuitive physics [44], imitation learning from videos [45], and grasp analysis [18].”
Pluralistic Image Completion (2019, methodology): “Perhaps most relevant are conditional VAEs (CVAE) [34, 37] and CVAE-GAN [2], but these were not specially targeted for image completion.”