Unsupervised Learning of Spatiotemporally Coherent Metrics

By Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, Yann LeCun. 2014

166 citations recorded by the source corpus.

Current state-of-the-art classification and detection algorithms train deep convolutional networks using labeled data. In this work we study unsupervised feature learning with convolutional networks in the context of temporally coherent unlabeled data. We focus on feature learning from unlabeled video data, using the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity priors. We establish a connection between slow feature learning and metric learning. Using this connection we define "temporal coherence" -- a criterion which can be used to set hyper-parameters in a principled and automated manner. In a transfer learning experiment, we show that the resulting encoder can be used to define a more semantically coherent metric without the use of labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Context Encoders: Feature Learning by Inpainting (2016, background): “Another way to use consistency is to track patches in frames of video containing task-relevant attributes and use the coherence of tracked patches to guide the training [39].”
Learning to See by Moving (2015, background): “Another line of work has focused on learning features that are invariant to transformations irrelevant to the identity of objects, either from images [14, 34], or from video [41].”
Learning Image Representations Tied to Ego-Motion (2015, methodology): “ed features vary slowly over continuous video, since visual stimuli can only gradually change between adjacent frames. Temporal coherence has been explored for unsupervised feature learning with CNNs [22, 37, 9, 3, 19], with applications to dimensionality reduction [10], object recognition [22, 37],”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “A number of prior works use temporal coherence [28, 29, 30, 31].”
Unsupervised Learning of Visual Representations Using Videos (2015, background): “Therefore, researchers have started focusing on learning feature representations using videos [11, 53, 27, 43, 56, 16, 48, 34, 45].”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “There are two common approaches for unsupervised learning: (a) using a discriminative framework with auxiliary tasks where supervision comes for free, such as context prediction [1,2] or temporal embedding [3,4,5,6,7,8]; (b) using a generative framework where the underlying model is compositional an”
Disentangling by Factorising (2018, background): “…or explicit knowledge about the true underlying factors of the data have excelled at disentangling (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Siddharth et al., 2017; Hinton et al., 2011; Mathieu et al., 2016; Goroshin et al., 2015; Hsu et al., 2017; Denton & Birodkar, 2017).”
Learning to Linearize Under Uncertainty (2015, background): “This work is more closely related to fully unsupervised approaches for learning representations from video such as [4][6][2][15][8].”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “s of data [11,10]. Self-supervised learning is another popular stream for learning invariant features. Visual invariance can be captured by the same instance/scene taken in a sequence of video frames [61,53,26,1,41,57,35,44,42,21]. For example, Wang and Gupta [61] leverage tracking of objects in vid”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, methodology): “While not rivaling supervised methods, this approach, and others that rely on temporal coherence, have achieved some measure of success, pointing to the potential of learning useful representations from video [15, 27, 30, 50, 55].”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, background): “For example, altering the natural temporal contiguity of visual stimuli hinders translation invariance in the inferior temporal cortex [27], and functions learned to preserve temporal coherence share behaviors observed in complex cells of the primary visual cortex [4].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “Such pretext tasks include ego-motion estimation [2, 24], tracking [49], temporal coherence [17], temporal order verification [36], and object motion mask prediction [39].”
Temporal Generative Adversarial Nets with Singular Value Clipping (2016, methodology): “Deep neural networks have also been applied to feature extraction from videos [51, 6, 48] in the same way as supervised learning.”
Unsupervised Learning using Sequential Verification for Action Recognition (2016, methodology): “Unlike our method, these works [26, 54, 58, 60] explicitly predict individual frames, but do not explore large image sizes or datasets.”
Learning by coincidence: Siamese networks and common variable learning (2018, methodology): “Siamese networks have been used to learn image features from consecutive frames in a video stream (single modality) in [10], [11] and [12], exploiting the similarity between consecutive frames; in this paper we discuss pairing through coincidence in general multimodal settings.”
Nonlinear ICA of Temporally Dependent Stationary Sources (2017, background): “Recent deep learning research (Mobahi et al., 2009; Springenberg and Riedmiller, 2012; Goroshin et al., 2015) as well as blind source separation research (Valpola and Karhunen, 2002; Hosseini and Deville, 2014) use similar ideas.”
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification (2016, methodology): “Unlike our method, these works [38, 43, 45, 47] explicitly predict individual frames, but do not explore large image sizes or datasets.”
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation (2017, background): “A number of prior works use temporal coherence [31, 11, 8, 14].”
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2016, background): “For example, recent works show that the temporal information from videos can be used as a plentiful source of supervision to learn embeddings that are useful for various tasks [14, 27].”