Unsupervised Feature Learning from Temporal Data

By Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, Yann LeCun. 2015

35 citations recorded by the source corpus.

Current state-of-the-art classification and detection algorithms rely on supervised training. In this work we study unsupervised feature learning in the context of temporally coherent video data. We focus on feature learning from unlabeled video data, using the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity. We establish a connection between slow feature learning to metric learning and show that the trained encoder can be used to define a more temporally and semantically coherent metric.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “The most well-known such criteria are image reconstruction [7, 38, 31, 29, 34, 23] (see [6] for a comprehensive overview) and feature slowness [48, 18].”
Learning to See by Moving (2015, methodology): “The quality of features learnt from this data were evaluated on four tasks - (1) Scene recognition on SUN [42] (section 5.1), (2) Visual odometery (section 5.3), (3) Keypoint matching (section 5.2) and (4) Object recognition on Imagenet [36] (section 5.4).”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “2015; Jayaraman andGrauman 2015) or track content across frames (Wang and Gupta 2015; Mobahi et al. 2009; Goroshin et al. 2015).”
Visually Indicated Sounds (2015, methodology): “Several methods have also used temporal proximity as a supervisory signal [33, 17, 47, 46].”
Spatiotemporal Residual Networks for Video Action Recognition (2016, background): “Spatiotemporal coherence is an important cue when working with time varying visual data and can be exploited to learn general representations from video in an unsupervised manner [7].”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “The idea of using a proxy task to learn a semantic feature embedding has been used in a number of works on self-supervised learning in computer vision (Agrawal et al., 2015; Doersch et al., 2015; Goroshin et al., 2015; Jayaraman & Grauman, 2015; Pathak et al., 2016; Wang & Gupta, 2015).”
Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA (2016, methodology): “Kernel-based methods [12, 26] and deep learning methods [23, 27, 9] have been developed to extend this principle to the general nonlinear case.”
Ambient Sound Provides Supervision for Visual Learning (2016, methodology): “With these methods, the supervisory signal may come from video, for example by having the algorithm estimate camera motion [1,17] or track content across frames [35,27,12].”
Object-Centric Representation Learning from Unlabeled Videos (2016, methodology): “Most prior work in this space produces a holistic image embedding, and attempts to learn feature representations for video frames as a whole [6,20,9,8,10].”
Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection (2017, methodology): “Our approach is motivated by the recent success of deep learning approaches (Goroshin et al. (2015); Madry et al. (2014); Lai et al. (2014)) in unsupervised feature learning for classiﬁcation and regression tasks.”
Learning Temporal Embeddings for Complex Video Analysis (2015, result): “Recent work from [5] also hints at a similar approach to exploit the slowness prior in videos.”
Semi-supervised tuning from temporal coherence (2015, methodology): “This is conceptually similar to the unsupervised feature learning proposed in Zou et al. (2011), Zou et al. (2012), Goroshin et al. (2015).”
Neural network-based clustering using pairwise constraints (2015, background): “The similar idea of leveraging temporal data is also presented in the report of Goroshin et al. (2015).”
Learning Temporal Regularity in Video Sequences (2016, unknown): “[17] trained a regularized high capacity (i.”
Unsupervised Learning of Visual Representations using Videos (2015, background): “Therefore, we have an update rule that contains two terms - a Hebbian term and an anti-Hebbian term. This can be interpreted as saying that if V is large compared to U then we are in a domain where preventing collapse is not a major concern, therefore the model’s learning is mostly antiHebbian and a”