Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

By William Lotter, Gabriel Kreiman, David D. Cox. 2016

1,007 citations recorded by the source corpus.

While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning - leveraging unlabeled examples to learn about the structure of a domain - remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network ("PredNet") architecture that is inspired by the concept of "predictive coding" from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Generating Videos with Scene Dynamics (2016, background): “Our work is also related to efforts to predict the future in video [33, 22, 43, 50, 42, 17, 8, 54] as well as concurrent work in future generation [6, 15, 20, 49, 55].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, positive): “This loss function is NCE (Sun et al., 2019a).”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Following the successes in learning distributed representations that efficiently encode the content of high-dimensional sensory data [45, 56, 76], recent work has focused on learning representations that are disentangled [6, 69, 68, 73, 71, 26, 27, 42, 10, 63, 16, 52, 53, 48, 9, 51].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Video prediction without actions has been studied for unstructured videos [33], [37], [38] and driving [39], [40].”
Deep visual foresight for planning robot motion (2016, methodology): “Video prediction is an active area of research in deep learning and computer vision [36], [41], [42] and, with the method described in this work, improvements to such predictive models could translate into improvements in robotic manipulation capabilities.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, methodology): “There have been a number of promising methods for frame prediction developed concurrently to this work [16].”
Unsupervised Representation Learning by Sorting Sequences (2017, background): “Recent approaches explore unlabeled video data to learn feature representation through egomotion [2, 14], order verification [25, 9], tracking [42], and future frame prediction [24].”
Stochastic Variational Video Prediction (2017, background): “A number of prior works have addressed video frame prediction while assuming deterministic environments (Ranzato et al., 2014; Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014; Lotter et al., 2017).”
Stochastic Video Generation with a Learned Prior (2018, background): “Lotter et al. (2016) propose a predictive coding model that learns features effective for recognition of synthetic faces, as well as predicting steering angles in the KITTI benchmark.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
Self-Supervised Visual Planning with Temporal Skip Connections (2017, background): “Action-conditioned video prediction has been explored in the context of synthetic video game images [12, 13] and robotic manipulation [14, 15, 16], and video prediction without actions has been studied for unstructured videos [17, 18, 19] and driving [20, 3].”
Video Representation Learning by Dense Predictive Coding (2019, methodology): “Previous approaches to future frame prediction in video [22, 23, 35, 40, 41] can roughly be divided into two types: those that predict a reconstruction of the actual frames [22, 23, 35, 41]; and those that only predict the latent representation (the embedding) of the frames [40].”
Stochastic Adversarial Video Prediction (2018, background): “The success of these methods can also be evaluated indirectly, such as evaluating the learned representations [42,43,44], or using the predictions for planning actions of a robotic system in the real world [45,4,46].”
Video-to-Video Synthesis (2018, methodology): “The video synthesis problem exists in various forms, including future video prediction [15,18,42,45, 50,65,68,71,77] and unconditional video synthesis [59,67,69].”
Learning to Generate Long-term Future via Hierarchical Prediction (2017, background): “Lotter et al. (2017) built an architecture inspired by the predictive coding concept in neuroscience literature that predicts realistic looking frames.”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “Instead of directly generating the pixels, transformation-based models focus on predicting the difference/transformation between frames and lead to sharper results [5, 8, 21, 35, 39, 40, 44, 45].”
End-to-End Learning of Driving Models from Large-Scale Video Datasets (2016, methodology): “This method is a special case of the more general task of video prediction; there are examples of video prediction models being applied to driving scenarios [4, 14].”