Learning to Linearize Under Uncertainty

By Ross Goroshin, Michaël Mathieu, Yann LeCun. 2015

145 citations recorded by the source corpus.

Training deep feature hierarchies to solve supervised learning tasks has achieved state of the art performance on many problems in computer vision. However, a principled way in which to train such hierarchies in the unsupervised setting has remained elusive. In this work we suggest a new architecture and loss for training deep feature hierarchies that linearize the transformations observed in unlabeled natural video sequences. This is done by training a generative model to predict video frames. We also address the problem of inherent uncertainty in prediction by introducing latent variables that are non-deterministic functions of the input into the network architecture.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Learning Features by Watching Objects Move (2016, background): “ing the input [4, 20, 44], predicting the pixels of the next frame in a video stream [17], metric learning on object track endpoints [46], temporally ordering shuffled frames from a video [29], and spatially ordering patches from a static image [8, 30].”
Deep multi-scale video prediction beyond mean square error (2015, methodology): “Goroshin et al. (2015) trained a convolutional network to learn to linearize motion in the code space and tested it on the NORB dataset.”
Stacked What-Where Auto-encoders (2015, background): “Recently, Goroshin et al. (2015) introduces a soft version of max and argmax operators within each pooling region:”
Dynamic Filter Networks (2016, methodology): “This issue could be alleviated with the methods proposed in [5, 6, 11].”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, methodology): “…bears some similarity with approaches such as Embed to Control [12], which learns states that follow a linear dynamic, or the approach of [11], [13], [14] which learn states that make possible to reconstruct the next observation with models such as PSRs (predictive state representation) [15].”
Deep Kalman Filters (2015, methodology): “Models that rely on predicting the next image based on the previous one (Goroshin et al. , 2015; Memisevic, 2013) may not sufﬁce to learn the structure of digits in the presence of large noise and rotation.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “…(Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hsieh et al., 2018; Fortuin et al., 2019; Deng et al., 2017; Goroshin et al., 2015).”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangling pose, style, or motion from content are classical vision tasks that has been studied with different degrees of supervision [72, 79, 80, 34, 19, 14, 21, 36].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018) or explicit inductive biases (Locatello et al.”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Deep spatial autoencoders for visuomotor learning (2015, methodology): “Some spatial network architectures have been proposed for tasks such as video prediction and dynamics modeling [30], [21].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “Much of the earlier work assumes some sort of supervision in terms of: (i) partial or full access to the generative factors per instance (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Karaletsos et al., 2015), (ii) knowledge about the nature of generative factors (e.g, translation, ro”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…et al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et al., 2015; Goroshin et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Whitney et al., 2016; Karaletsos et al., 2016; Chen et al., 2016; Higgins et al.,…”
Stochastic Video Generation with a Learned Prior (2018, background): “Henaff et al. (2017) disentangle deterministic and stochastic components of a video by encoding prediction errors of a deterministic model in a low dimensional latent variable.”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Here, we explore another potential principle for exploiting video for unsupervised learning: prediction of future image frames [16, 28, 32, 33, 34, 46, 48].”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, background): “Several recent papers [5, 41, 14] have trained unsupervised image representations targeting specific narrow task s.”