Watching the World Go By: Representation Learning from Unlabeled Videos

By Daniel Gordon, Kiana Ehsani, Dieter Fox, Ali Farhadi. 2020

96 citations recorded by the source corpus.

Recent single image unsupervised representation learning techniques show remarkable success on a variety of tasks. The basic principle in these works is instance discrimination: learning to differentiate between two augmented versions of the same image and a large batch of unrelated images. Networks learn to ignore the augmentation noise and extract semantically meaningful representations. Prior work uses artificial data augmentation techniques such as cropping, and color jitter which can only affect the image in superficial ways and are not aligned with how objects actually change e.g. occlusion, deformation, viewpoint change. In this paper, we argue that videos offer this natural augmentation for free. Videos can provide entirely new views of objects, show deformation, and even connect semantically similar but visually distinct concepts. We propose Video Noise Contrastive Estimation, a method for using unlabeled video to learn strong, transferable single image representations. We demonstrate improvements over recent unsupervised single image techniques, as well as over fully supervised ImageNet pretraining, across a variety of temporal and non-temporal tasks. Code and the Random Related Video Views dataset are available at this https URL

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[48] Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi.”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving (2021, background): “More recently, contrastive learning is integrated to learn temporal changes [18, 63].”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Several recent ones [28, 35, 3, 68, 2, 71, 92, 62] relate to image-based approaches [36, 8, 12, 89].”
In the Wild: From ML Models to Pragmatic ML Systems (2020, methodology): “We use the models from VINCE [12] for the MoCo [18] self-supervised ImageNet-1K pretrained models of ResNet18 and ResNet50.”
Space-Time Correspondence as a Contrastive Random Walk (2020, methodology): “Strong image-level visual representations can be learned by heuristically choosing views that are close in space [80, 31, 13], in time [70, 30, 25] or both [35, 77], even when relying on noisy negative samples.”
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data (2021, methodology): “Other methods use metadata such as hashtags [23, 40, 27], geolocation [45] or the video structure [14] as a source of noisy supervision.”
Are We Overfitting to Experimental Setups in Recognition (2020, methodology): “Unsupervised learning methods like autoencoders [45] and more recent self-supervision methods [21, 33, 12] like Momentum Contrast (MoCo) [17] and SimCLR [5] have begun to produce representations as rich as that of supervised learning and achieve similar accuracy on various downstream tasks.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”