Unsupervised Learning of Visual Representations Using Videos

By X. Wang, Abhinav Kumar Gupta. 2015

229 citations recorded by the source corpus.

Is strong supervision necessary for learning a good visual representation? Do we really need millions of semantically-labeled images to train a Convolutional Neural Network (CNN)? In this paper, we present a simple yet surprisingly powerful approach for unsupervised learning of CNN. Specifically, we use hundreds of thousands of unlabeled videos from the web to learn visual representations. Our key idea is that visual tracking provides the supervision. That is, two patches connected by a track should have similar visual representation in deep feature space since they probably belong to same object or object part. We design a Siamese-triplet network with a ranking loss function to train this CNN representation. Without using a single image from ImageNet, just using 100K unlabeled videos and the VOC 2012 dataset, we train an ensemble of unsupervised networks that achieves 52% mAP (no bounding box regression). This performance comes tantalizingly close to its ImageNet-supervised counterpart, an ensemble which achieves a mAP of 54.4%. We also show that our unsupervised network can perform competitively in other tasks such as surface-normal estimation.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
Contrastive Multiview Coding (2019, background): “A powerful family of models for unsupervised representations are collected under the umbrella of “self-supervised” learning [64, 35, 85, 84, 78, 60, 83].”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “In Computer Vision [28] use a triplet loss on tracked video patches so that patches from the same object at different timesteps are more similar to each other than to random patches.”
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (2016, methodology): “– Repurpose the CFN,[2] and [4] to classification on ImageNet 2012.”
Colorful Image Colorization (2016, methodology): “We test how well our model performs in generalization tasks, compared to previous [14,8,15,10] and concurrent [16] self-supervision algorithms, and find that our method performs surprisingly well, achieving state-of-the-art performance on several metrics.”
On Mutual Information Maximization for Representation Learning (2019, background): “One possible avenue is to consider extensions to MI which explicitly account for the modeling power and computational constraints of the observer, such as the recently introduced F-information Xu et al. (2020). Alternatively, one can investigate other statistical divergences to measure the discrepan”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Videos can provide information about how objects move [54], the relation between viewpoints [69, 70] etc.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “(Wang and Gupta, 2015) adopt Siamese-triplet network to rank continuous patches more similar than patches of different videos.”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, positive): “While there has been much work on learning from unlabeled image data (Doersch et al 2015; Wang and Gupta 2015; Le et al 2012), an audio signal may provide information that is largely orthogonal to that available in images alone – information about semantics, events, and mechanics are all readily…”
Are Labels Necessary for Neural Architecture Search? (2020, neutral): “, [10,9,31,35,22,12,23,32,30,13].”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “[42], with representation size 1024, is trained on 100K videos from YouTube for unsupervised patch triplet ranking.”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, methodology): “Recent work uses CNNs to explore the power of learning slow features, also referred to as “temporally coherent” features [31, 3, 47, 13, 42].”
Cross and Learn: Cross-Modal Self-Supervision (2018, background): “Approaches which use video data mainly exploit temporal context as source of supervision [40,24,4,5,2].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “redict the correct ordering of frames. [11] exploit both temporal and spatial co-occurrence statistics to learn visual groups. [12] extend slow feature analysis using higher order temporal coherence. [13] track patches in a video to supervise their embedding via a triplet loss while [14] do the same”
Self-Supervised Representation Learning by Rotation Feature Decoupling (2019, background): “Another different but popularly adopted strategy is to discover supervisory signals in videos like tracking imagepatches [47] and sorting frame sequences [30].”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Various types of self-supervised signals have been used to learn strong visual representations, such as spatial arrangement [44], contextual information [14, 68], color [37, 64], the arrow of time [70, 75, 23, 38, 41, 17], future prediction [40, 63, 76, 45], consistency inmotion [3, 26], viewsynthes”
ShapeCodes: Self-supervised Feature Learning by Lifting Views to Viewgrids (2017, methodology): “Our work relates to a growing body of work on self-supervised representation learning [7, 8, 9, 10, 11, 12, 13, 14, 15, 16], where “pretext” tasks requiring no manual labels are posed to the feature learner to instill visual concepts useful for recognition.”
Manipulating Attributes of Natural Scenes via Hallucination (2018, methodology): “To address this issue, we propose to selectively sample mismatched layouts for a given real image, inspired by the practice of hard negative mining [Wang and Gupta 2015].”