Anticipating the future by watching unlabeled video

By Carl Vondrick, Hamed Pirsiavash, Antonio Torralba. 2015

146 citations recorded by the source corpus.

In many computer vision applications, machines will need to reason beyond the present, and predict the future. This task is challenging because it requires leveraging extensive commonsense knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently obtaining this knowledge is through the massive amounts of readily available unlabeled video. In this paper, we present a large scale framework that capitalizes on temporal structure in unlabeled video to learn to anticipate both actions and objects in the future. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. We experimentally validate this idea on two challenging "in the wild" video datasets, and our results suggest that learning with unlabeled videos significantly helps forecast actions and anticipate objects.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep multi-scale video prediction beyond mean square error (2015, methodology): “We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset.”
Visually Indicated Sounds (2015, methodology): “Several methods have also used temporal proximity as a supervisory signal [33, 17, 47, 46].”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “Prediction has been an active area of research in computer vision, including prediction of future images [9, 10, 11], motion [12], and even physical events [13, 14, 15].”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “The higher layers of a deep neural networks trained on the ImageNet challenge provide one such feature space (Vondrick et al., 2015).”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, methodology): “Prior methods generally reconstruct frames from the internal state of the model, and some predict the internal state directly, without producing images [23].”
Learning to Linearize Under Uncertainty (2015, background): “For instance, [13] assumes that the pre-trained network from [7] is already available and training consists of learning to mimic this network.”
Stochastic Variational Video Prediction (2017, background): “A number of prior works have addressed video frame prediction while assuming deterministic environments (Ranzato et al., 2014; Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014; Lotter et al., 2017).”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, background): “Therefore, the vision community has recently pushed back and directly attacked the problem of full blown visual prediction: recent works have proposed predicting pixels [6,7] or the high dimensional fc7 features [8] themselves.”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “In order to obtain features one can, for example, first train a neural network on still image object-recognition tasks [35].”
Playing hard exploration games by watching YouTube (2018, methodology): “This idea is motivated by several recent works in the self-supervised feature learning literature [3, 11, 12, 28, 42, 45, 26].”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, background): “Thus, representations useful for recognizing high-level concepts are also likely to possess this property of “slowness”.”
Learning Temporal Transformations from Time-Lapse Videos (2016, background): “Given large amounts of unlabeled video data from the internet, [15] trains a deep network to predict visual representations of future images, enabling them to anticipate both actions and objects before they appear.”
Unsupervised Learning using Sequential Verification for Action Recognition (2016, background): “[62, 63] also consider the task of predicting the future from videos, but consider it as their end task and do not use it for unsupervised pre-training.”
Grounded Human-Object Interaction Hotspots From Video (2018, methodology): “Future prediction has been applied to action anticipation [26, 55, 32, 44], active-object forecasting [16], and to guide demonstration learning in robotics [13, 14, 11].”
The Pose Knows: Video Forecasting by Generating Pose Futures (2017, background): “Activity Forecasting: Much work in activity forecasting has focused on predicting future semantic action classes [14, 24, 26] or more generally semantic information [36, 46].”
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification (2016, background): “[48, 49] also consider the task of predicting the future from videos, but consider it as their end task and do not use it for unsupervised pre-training.”
Transformation-based Adversarial Video Prediction on Large-Scale Data (2020, background): “Finally, another line of work reformulates video prediction in other feature spaces than the pixel space, such as high level image features [63, 68], optical flow [72], dense trajectories [73] and segmentation maps [46, 47].”
Unsupervised incremental learning of deep descriptors from video streams (2017, methodology): “This idea was suggested by [11], but was essentially applied with some suc cess to predict future frames with unsupervised feature learn ing, [12] among the most notable experiments.”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, background): “Early work on prediction of future video frames focused on deterministic predictive models (Ranzato et al., 2014; Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014).”