Learning Image Representations Tied to Ego-Motion

By Dinesh Jayaraman, Kristen Grauman. 2015

254 citations recorded by the source corpus.

Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images. We propose to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video. Specifically, we enforce that our learned features exhibit equivariance, i.e, they respond predictably to transformations associated with distinct ego-motions. With three datasets, we show that our unsupervised feature learning approach significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks. In the most challenging test, we show that features learned from video captured on an autonomous driving platform improve large-scale scene recognition in static images from a disjoint domain.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “For example, camera motion [1, 32], tracking [61], text proximity [41], and other cues [11, 62, 30, 60, 29] can be used to ﬁnd examples of features that should be ‘similar’ or predictable from one another, whereas negative examples are usually sampled randomly.”
Learning Features by Watching Objects Move (2016, background): “% m e a n A P Object detection (VOC 2012): Full train set ImageNet [21] Tracking-gray [43] Colorization [48] Context [6] BiGAN [8] Sounds [30] Sup.”
Context Encoders: Feature Learning by Inpainting (2016, background): “Most closely related to the present paper are efforts at exploiting spatial context as a source of free and plentiful supervisory signal.”
Colorful Image Colorization (2016, methodology): “resentations in this way goes back at least to autoencoders [6]. More recent works have explored feature learning via data imputation, where a held-out subset of the complete data is predicted (e.g., [7,8,9,10,11,12,13]). Our method follows in this line, and can be termed a cross-channel encoder. We”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “, foreground-background segmentation via video segmentation [43], optical-ﬂow [12,33], future-frame synthesis [49], audio prediction from video [5,42], audiovideo alignment [2], ego-motion estimation [23], slow feature analysis with higher order temporal coherence [24], transformation between frames”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, background): “Other meth-ods generate pseudo-labels from egomotion [1, 18], video tracking [38], inpainting [28], co-occurence [17], puzzle-solving [25], color [42, 23], and sound [27].”
Generating Videos with Scene Dynamics (2016, background): “Finally, this paper is related to a growing body of work that capitalizes on large amounts of unlabeled video for visual recognition tasks [18, 47, 37, 13, 24, 25, 3, 32, 26, 27, 19, 41, 42, 1].”
Representation Learning by Learning to Count (2017, background): “Other recent works use as surrogate labels ego-motion [1, 17], temporal ordering in video [27, 32], sound [31], and physical interaction [34].”
Unsupervised Learning by Predicting Noise (2017, background): “Others have shown that temporal coherence in videos also provides a signal that can be used to learn powerful visual features (Agrawal et al., 2015; Jayaraman & Grauman, 2015; Wang & Gupta, 2015).”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, background): “We show through numerous experiments that this combination leads to a competitive semi-supervised learning method.”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “With these methods, the supervisory signal may come from video, for example by having the algorithm estimate camera motion (Agrawal et al 2015; Jayaraman and Grauman 2015) or track content across frames (Wang and Gupta 2015; Mobahi et al 2009; Goroshin et al 2015).”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “Some examples of these auxiliary tasks include predicting: the relative location of two patches [2], ego-motion in videos [15,16], physical signals [17,18,19].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “…audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames…”
Network Dissection: Quantifying Interpretability of Deep Visual Representations (2017, methodology): “For self-supervised training tasks, we select several recent models trained on predicting context (context) [9], solving puzzles (puzzle) [21], predicting ego-motion (egomotion) [14], learning by moving (moving) [1], predicting video frame order (videoorder) [18] or tracking (tracking) [33], detecti”