Learning to See by Moving

By Pulkit Agrawal, João Carreira, Jitendra Malik. 2015

571 citations recorded by the source corpus.

The current dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images. Is it also possible to learn features for a diverse set of visual tasks using any other form of supervision? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world. Drawing inspiration from this observation, in this work we investigated if the awareness of egomotion(i.e. self motion) can be used as a supervisory signal for feature learning. As opposed to the knowledge of class labels, information about egomotion is freely available to mobile agents. We found that using the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt using class-label as supervision on the tasks of scene recognition, object recognition, visual odometry and keypoint matching.

Subfields: Aerospace Engineering, Biophysics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Biophysics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Non-visual information can be equally powerful: information about camera motion (Agrawal et al., 2015; Jayaraman & Grauman, 2015), scene geometry (Zamir et al.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “[1] make use of egomotion information obtained by odometry sensors mounted on a vehicle to pre-train a CNN model.”
Unsupervised Representation Learning by Predicting Image Rotations (2018, methodology): “…(2016a) and Larsson et al. (2016) train ConvNets to colorize gray scale images, Doersch et al. (2015) and Noroozi & Favaro (2016) predict the relative position of image patches, and Agrawal et al. (2015) predict the egomotion (i.e., self-motion) of a moving vehicle between two consecutive frames.”
Learning Features by Watching Objects Move (2016, background): “We compare our approach to ConvNet representations produced by recent prior work on unsupervised learning [2, 8, 10, 30, 33, 35, 46, 51].”
Context Encoders: Feature Learning by Inpainting (2016, background): “Most closely related to the present paper are efforts at exploiting spatial context as a source of free and plentiful supervisory signal.”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (2016, methodology): “[21] has been applied to [9,34,2] with a remarkable increase in performance for object detection on PASCAL VOC 2007.”
Colorful Image Colorization (2016, methodology): “U) ne-tuned layers fc8 fc6-fc8 all all all ImageNet [38] 76.8 78.9 79.9 56.8 48.0 Gaussian { { 53.3 43.4 19.8 Autoencoder 24.8 16.0 53.8 41.9 25.2 k-means [38] 32.0 39.2 56.6 45.6 32.6 Agrawal et al. [8] 31.2 31.0 54.2 43.9 { Wang & Gupta [15] 28.1 52.2 58.7 44.0 { *Doersch et al. [14] 44.7 55.1”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “In Table 5, we explore how well our perceptual task correlates to semantic tasks on the PASCAL dataset [15], using results summarized in [63], including additional selfsupervised methods [1, 43, 12, 56, 62, 41].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “, 2016), audio-video alignment (Arandjelović & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Agrawal et al., 2015) and patch tracking in videos (Wang & Gupta,”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, background): “Some attempts also use video information to provide weak supervision to learn feature representations [1, 44].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “It has been widely used in the video domain [39, 40], the robotics domain [41, 42] and the image domain [24, 35].”
Multi-task Self-Supervised Visual Learning (2017, background): “In the last few years, a considerable number of such tasks have been proposed [1, 2, 6, 7, 8, 17, 20, 21, 23, 25, 26, 27, 28, 29, 31, 39, 40, 42, 43, 46, 47], such as asking a neural network to colorize grayscale images, ﬁll in image holes, solve jigsaw puzzles made from image patches, or predict…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, background): “Other meth-ods generate pseudo-labels from egomotion [1, 18], video tracking [38], inpainting [28], co-occurence [17], puzzle-solving [25], color [42, 23], and sound [27].”
Adversarial Feature Learning (2016, methodology): “We report results on each of these tasks in Table 3, comparing BiGANs with contemporary approaches to unsupervised [4, 17] and weakly supervised [1, 30] feature learning in the visual domain, as well as the baselines discussed in Section 4.”
Data-dependent Initializations of Convolutional Neural Networks (2015, positive): “Agrawal et al. (2015) uses egomotion, as measured by a moving car in a city to pre-train a model. While this information is not always readily available, it can be read from sensors and is thus “free.” We believe egomotion information does not often correlate with the kind of semantic information th”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Others leverage the temporal signal available in videos by predicting the camera transformation between consecutive frames [1], exploiting the temporal coherence of tracked patches [63] or segmenting video based on motion [45].”