Learning Features by Watching Objects Move

By Deepak Pathak, Ross B. Girshick, Piotr Doll'ar, Trevor Darrell, Bharath Hariharan. 2016

539 citations recorded by the source corpus.

This paper presents a novel yet intuitive approach to unsupervised feature learning. Inspired by the human visual system, we explore whether low-level motion-based grouping cues can be used to learn an effective visual representation. Specifically, we use unsupervised motion-based segmentation on videos to obtain segments, which we use as pseudo ground truth to train a convolutional network to segment objects from a single frame. Given the extensive evidence that motion plays a key role in the development of the human visual system, we hope that this straightforward approach to unsupervised learning will be more effective than cleverly designed pretext tasks studied in the literature. Indeed, our extensive experiments show that this is the case. When used for transfer learning on object detection, our representation significantly outperforms previous unsupervised approaches across multiple settings, especially when training data for the target task is scarce.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, background): “Self-supervised learning approaches have seen significant interest in computer vision, often focusing on different pretext tasks for pre-training [15, 50, 35, 59, 38, 17].”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “, transformations of a single (“exemplar”) image [17], patch orderings [13, 45], tracking [59] or segmenting objects [47] in videos, or clustering features [3, 4].”
Rethinking ImageNet Pre-Training (2018, background): “, for selfsupervised learning [5, 47, 33, 32]), as now we learn that even random initialization could produce excellent results.”
Exploring the Limits of Weakly Supervised Pretraining (2018, background): “There are two established protocols for judging the quality of a pretrained model (see [29] §3 for a discussion).”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “We further compare against three prior works that learn image representations from video data: The motion segmentation (MS) [47] and the multi-task SSL (MTSSL) models from [13], and the transitive invariance (TI) model from [63].”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “While work within the vision community has shown increasing promise [21–23], the greatest impact of self-supervised learning so far is through language models like ELMo [13], BERT [12], and GPT [14] which have set new high-water marks on many NLP tasks.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, neutral): “, 2016), and object boundary cues (Li et al., 2016; Pathak et al., 2016).”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “[99] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan.”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “For videos, self-supervision strategies include: leveraging temporal continuity via tracking [44, 45], predicting future [42], or preserving the equivariance of egomotion [13, 50, 30].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “Speciﬁcally, we ﬁrst train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Coloriza”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “We additionally evaluate self-supervised methods, including puzzle-solving [40], cross-channel prediction [62, 63], learning from video [42], and generative modeling [13].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Large Scale Adversarial Representation Learning (2019, methodology): “) Compared with a number of modern selfsupervised approaches [25, 3, 34, 32, 9, 14] and combinations thereof [4], our BigBiGAN approach based purely on generative models performs well for representation learning, state-of-the-art among recent unsupervised learning results, improving upon a recently ”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “, 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-flow (Gan et al.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “[55, 49, 14, 75]) do not benefit significantly from increasing depth.”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “Some prominent examples are image inpainting [37], image colorization [47], its improved variant split-brain [48] and motion segmentation prediction [36].”
Multi-task Self-Supervised Visual Learning (2017, methodology): “We follow Pathak et al. [27], except that we replace their tracking algorithm with Improved Dense Trajectories [41].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Videos can provide information about how objects move [54], the relation between viewpoints [69, 70] etc.”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “We hypothesize that for some tasks, especially those with inherent uncertainty in the prediction, the classification loss may lead to better representations as well, as the network will be incentivized to match the whole distribution, and not only predict the first moment.”