Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

By Senthil Purushwalkam, Abhinav Kumar Gupta. 2020

238 citations recorded by the source corpus.

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “Previous works show that transformations altering content C can even be detrimental, which is referred to as aggressive augmentations in [32].”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The irregular representation space raises the risk of degeneration [37, 55] and often necessitates sophisticated post-adjustment or ﬁne-tuning to improve the sequence representations [16, 30, 32, 60].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Video can help supervise the learning of image representations [67, 48, 64, 23, 52], correspondences [68, 14], and robotic behaviors [55] thanks to its rich content about different views of objects and its motion and tracking cues.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “That is, the rendering pipeline offers complete control over (hereto-fore) fixed design choices such as camera intrinsics, scene lighting, object-centeredness [45], the level of “photogra-pher’s bias” [6], data domain, and so on.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, background): “, [16, 41]), which could lead to conflicting clues [71] and over-fitting to a specific combination of augmentations [94].”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, background): “Some of the most effective transformations, like cropping, rely on the images being object centric [21].”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “A set of other work [2, 90, 110, 99, 91, 75, 95] focuses on providing empirical and theoretical understanding of contrastive learning.”
Understanding the Behaviour of Contrastive Loss (2020, background): “Purushwalkam et al [22] try to de-mystify the unsupervised contrastive learning by focusing on the relation of data augmentation and the corresponding invariances.”
Contrastive Learning with Hard Negative Samples (2020, methodology): “Positive sampling techniques have also been proposed for sentence, audio, and video data (Logeswaran & Lee, 2018; Oord et al., 2018; Purushwalkam & Gupta, 2020; Sermanet et al., 2018).”
Hard Negative Mixing for Contrastive Learning (2020, background): “In an interesting recent study, Purushwalkam and Gupta [52] study the robustness of contrastive self-supervised learning methods like MoCo [30] and PIRL [46] and saw that despite the fact that they learn occlusion-invariant representations, they fail to capture viewpoint and category instance…”
A Survey on Contrastive Self-supervised Learning (2020, background): “rastive learning has decreased the gap in performance with supervised models, there is a need for more theoretical analysis to form a solid justiﬁcation. For instance, a study by Purushwalkam et. al. [82] reveals that approaches like PIRL [17] and MoCo [14] fail to capture viewpoint and category ins”
A Cookbook of Self-Supervised Learning (2023, background): “…such as MoCo [He et al., 2020b] and PIRL [Misra and Maaten, 2020], learn occlusion invariance since they are trained with random crops on ImageNet where foreground objects are often large so that different crops contain different parts of the same object [Purushwalkam and Gupta, 2020].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Numerous CL-based methods are available in the literature [89]–[109].”
Are We Overfitting to Experimental Setups in Recognition (2020, methodology): “Unsupervised learning methods like autoencoders [45] and more recent self-supervision methods [21, 33, 12] like Momentum Contrast (MoCo) [17] and SimCLR [5] have begun to produce representations as rich as that of supervised learning and achieve similar accuracy on various downstream tasks.”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,78,85,60,84,58,57,21,51,86,66,22,48,91,16,87,70,45], and contrastive objectives with visual [74, 79, 34, 53, 28, 92] and”