A Simple Framework for Contrastive Learning of Visual Representations
By Ting-Li Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. Hinton. International Conference on Machine Learning, 2020
25,300 citations recorded by the source corpus.
This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/2002.05709
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Representation & Self-Supervised Learning
- Ishan Misra
- Trevor Darrell
- Yoshua Bengio
- Andrea Vedaldi
- Andrew Zisserman
- Dacheng Tao
- Yann LeCun
- Quoc V. Le
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or unified text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “SimCLRv2 The SimCLRv2 (Chen et al., 2020d) project released pre-trained and fine-tuned models in various settings.”
- Masked Autoencoders Are Scalable Vision Learners (2021, background): “Contrastive and related methods strongly depend on data augmentation [7, 20, 8].”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, positive): “As an example, “MoCo v2” [8], an extension of a prelim-inary version of this manuscript, achieves 71.1% accuracy with R50 (up from 60.6%), given small changes on the data augmentation and output projection head [7].”
- Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “…accuracy with a ResNet-50 model after SSL pre-training for 1000 epochs on the ImageNet dataset and using the linear evaluation protocol, freezing the pre-trained feature extractor, and fine-tuning a linear classifier on top for 60 epochs using the full ImageNet dataset with all labels [118].”
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Other similar loss functions are also explored [24].”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020; He et al., 2020).”
- On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “We include self-supervised SimCLR [6] (variants: linear classifier on fixed representation (lin), fine-tuned on 10% (ft-10), and 100% (ft-100) of the IMAGENET data), and self-supervised-semi-supervised (S4L) [71] models that have been fine-tuned to 10% and 100% of the IMAGENET data.”
- Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “We run experiments with two popular image-only contrastive visual representation learning methods: SimCLR (Chen et al., 2020a) and MoCo v2 (Chen et al., 2020b).”
- Learning Visual Representations with Caption Annotations (2020, methodology): “they require either extremely large training sets (up to a hundred million images) [7,24] or need to be trained much longer with larger networks to express their full potential [9,26].”
- Self-Supervised MultiModal Versatile Networks (2020, methodology): “use a contrastive loss and nonlinear projection heads [16]).”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “In SimCLR [1], the MLP projection head g(·) is discarded entirely after pretraining, while only the ResNet encoder f(·) is used during the fine-tuning.”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Most unsupervised methods for representation learning can be categorized as either generative or discriminative [23, 8].”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “1: Comparison of pretraining tasks for learning visual representations: Contrastive methods used for self-supervised learning [20, 23, 25, 26] provide a semantically sparse learning signal, encouraging different transforms of an image to have similar features.”
- Improved Baselines with Momentum Contrastive Learning (2020, methodology): “…., Momentum Contrast (MoCo) [6] shows that unsupervised pre-training can surpass its ImageNet-supervised counterpart in multiple detection and segmentation tasks, and SimCLR [2] further reduces the gap in linear classifier performance between unsupervised and supervised pre-training representations.”
- Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “A large body of work on self-supervised learning focuses on discriminative approaches coined instance classification [12, 20, 33, 73], which considers each image a different class and trains the model by discriminating them up to data augmentations.”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “In this regime, the larger batch improves accuracy thanks to more negative samples [19, 9].”
- Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “These methods discriminate either between each instance feature [8, 22, 37] or between their cluster assignments [2, 7, 32].”