Unsupervised Learning of Visual Structure using Predictive Generative Networks

By William Lotter, Gabriel Kreiman, David D. Cox. 2015

133 citations recorded by the source corpus.

The ability to predict future states of the environment is a central pillar of intelligence. At its core, effective prediction requires an internal model of the world and an understanding of the rules by which the world changes. Here, we explore the internal models developed by deep neural networks trained using a loss based on predicting future frames in synthetic video sequences, using a CNN-LSTM-deCNN framework. We first show that this architecture can achieve excellent performance in visual sequence prediction tasks, including state-of-the-art performance in a standard 'bouncing balls' dataset (Sutskever et al., 2009). Using a weighted mean-squared error and adversarial loss (Goodfellow et al., 2014), the same architecture successfully extrapolates out-of-the-plane rotations of computer-generated faces. Furthermore, despite being trained end-to-end to predict only pixel-level information, our Predictive Generative Networks learn a representation of the latent structure of the underlying three-dimensional objects themselves. Importantly, we find that this representation is naturally tolerant to object transformations, and generalizes well to new tasks, such as classification of static images. Similar models trained solely with a reconstruction loss fail to generalize as effectively. We argue that prediction can serve as a powerful unsupervised loss for learning rich internal representations of high-level object features.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, methodology): “[22] use adversarial loss in an LSTM network for visual sequence prediction.”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, methodology): “Many have suggested that temporal experience with objects as they move and undergo transformations can serve as an important signal for learning about the structure of objects [1, 10, 13, 15, 26, 28, 30, 32, 33, 38, 46, 48, 55, 57, 58].”
Toward an Integration of Deep Learning and Neuroscience (2016, methodology): “Alternatively, explicit cost functions could be computed, delivered to an optimizing network, and used for unsupervised learning, following a variety of principles being discovered in machine learning (e.g., Radford et al., 2015; Lotter et al., 2015).”
Perceptual Adversarial Networks for Image-to-Image Transformation (2017, background): “In addition, these kind of works include, but not limited to, the PGN [45] for video prediction, the SRGAN [25] for super-resolution, the ID-CGAN for image de-raining [24], the iGAN [46] for interactive application, the IAN [47] for photo modiﬁcation, and the Context-Encoder for image in-painting…”
Adversarial Transfer Learning (2018, methodology): “If this loss function is used, the trained generator would produce blurry images [76] as shown in Figure 5.”
Global-Local Face Upsampling Network (2016, methodology): “Since then, GANs have been successfully used for image [34,35], scene [36], and sequence synthesis [37] tasks.”
Deep reverse tone mapping (2017, methodology): “This approach was recently adopted for video prediction [Lotter et al. 2016; Srivastava et al. 2015].”
Decomposing Motion and Content for Natural Video Sequence Prediction (2017, methodology): “There have been a number of recent attempts on frame prediction (Srivastava et al., 2015; Mathieu et al., 2015; Oh et al., 2015; Goroshin et al., 2015; Lotter et al., 2015; Ranzato et al., 2014), ∗This work was done while SH and XL were visiting the University of Michigan.”
Spatio-Temporal AutoEncoder for Video Anomaly Detection (2017, methodology): “Inspired by the previous works [11, 15, 18, 25, 29] which prove that predictive networks benefit learning video representations, we design a prediction branch in the decoder part to predict the future T frames after the input video clip.”
An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos (2018, methodology): “Similar efforts were performed in [56] using a CNN-LSTM-deCNN framework while combining MSE and an adversarial loss.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “[276] show an excellent description of the significance of being capable of modeling multi-modal data.”
End-to-End Adversarial Retinal Image Synthesis (2018, background): “, (rn, vn)} will produce low-quality results with lack of detail [13], due to the model selecting an average of many potentially valid representations.”
SimVP: Simpler yet Better Video Prediction (2022, unknown): “2014-2015 [41, 55, 71] [36, 46] [40] 2016-2017 [38, 47, 63, 67] [3, 9, 15, 32, 62] [20, 35, 59] 2018-2019 [22, 43, 56, 65, 69, 74] [6, 10, 61, 66, 73] [70] [16, 29, 72] 2020-2021 [68] [18, 23] [48] [7, 54]”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “GANs [100]: are the backbone of many video prediction approaches [43], [49]–[55], [57], [65], [67], [68], [78], [101]– [106].”