A Simple Framework for Contrastive Learning of Visual Representations

By Ting-Li Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. Hinton. International Conference on Machine Learning, 2020

25,300 citations recorded by the source corpus.

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

View Paper: https://arxiv.org/abs/2002.05709

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yoshua Bengio
  4. Quoc V. Le
  5. Sergey Levine
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Dacheng Tao

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yi Yang

In Representation & Self-Supervised Learning

  1. Ishan Misra
  2. Trevor Darrell
  3. Yoshua Bengio
  4. Andrea Vedaldi
  5. Andrew Zisserman
  6. Dacheng Tao
  7. Yann LeCun
  8. Quoc V. Le

Citations in the corpus