Supervised Contrastive Learning

By Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, Dilip Krishnan. 2020

6,562 citations recorded by the source corpus.

Cross entropy is the most widely used loss function for supervised training of image classification models. In this paper, we propose a novel training methodology that consistently outperforms cross entropy on supervised learning tasks across different architectures and data augmentations. We modify the batch contrastive loss, which has recently been shown to be very effective at learning powerful representations in the self-supervised setting. We are thus able to leverage label information more effectively than cross entropy. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. In addition to this, we leverage key ingredients such as large batch sizes and normalized embeddings, which have been shown to benefit self-supervised learning. On both ResNet-50 and ResNet-200, we outperform cross entropy by over 1%, setting a new state of the art number of 78.8% among methods that use AutoAugment data augmentation. The loss also shows clear benefits for robustness to natural corruptions on standard benchmarks on both calibration and accuracy. Compared to cross entropy, our supervised contrastive loss is more stable to hyperparameter settings such as optimizers or data augmentations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [28, 2, 9, 3, 16, 35, 26, 5, 8, 10, 15, 21, 34, 39] for visual representations have been proposed.”
Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “1 left part, supervised learning [30] on image-label data typically aims at mapping images to discrete labels, and completely ignores the textual concept associated with each label during the training.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “[32].”
What makes for good views for contrastive learning (2020, background): “Leveraging labeled data in contrastive representation learning has been shown to guide representations towards task-relevant features that improve performance [61, 25, 29, 57].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “However, they discriminate instances either unsupervisedly or with image classes [46], whereas we perform a pixel-wise instance discrimination task, as annotated by panoptic segmentation ground truth.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “It is based on data and class label augmentation and supervised contrastive learning [72].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “In particular: i) it updates the logits contained in the memory buffer by implanting novel secondary information made available for past classes; ii) it exploits supervised contrastive learning [40] to prepare the model for future tasks.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, neutral): “…[6] 79.75 ± 0.84* DyTox [14] 79.21 ± 0.10 L2P [66] 81.07 ± 0.13* EWC [32] 0/class 49.19 SimCLR [10]-CaSSLe [18] 0/class 44.2* BYOL [20]-CaSSLe [18] 49.7* Barlow Twins[72]-CaSSLe [18] 48.9* Supervised Contrastive [30]-CaSSLe [18] 50.9* L2P [66] 40 image/language prompting for S-iPrompts/S-liPrompts.”
Online Continual Learning with Contrastive Vision Transformer (2022, background): “Inspired by recent breakthroughs in self-supervised learning [16,11,30,72], we find that the knowledge learned by supervised contrastive learning [31,41,28] reveals greater robustness and transferability.”
Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, methodology): “Step 2 [10,11,12,13,14,15,16,17,18,19] Kettle, Folder, Keyboard, Flipflops, Pencil, Bed, Hammer, ToothBrush, Couch, Bike”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “Contrastive learning has also proved itself to be extremely competitive in supervised settings [25].”
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, methodology): “As well they are becoming a popular alternative to cross-entropy (CE) in the supervised setting [12, 21], referred to as SupCon.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, methodology): “In more detail, we always use Stochastic Gradient Descent as optimizer and a number of epochs that varies according to the dataset (50 for CIFAR-100, 80 for miniImageNetand 70 for NTU); in line with [11], [31], [41], we also define a set of epochs at which the learning rate is divided by 10 ([35,45]”
Self-Supervised Models are Continual Learners (2021, background): “After the popularization of contrastive learning [8, 14] for unsupervised learning of representations, [18] proposed a supervised version of the contrastive loss.”
GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, positive): “We also include a supervised representation learning loss [19] in our CL objective, enhancing the learned representations over the model’s lifetime.”
Co2L: Contrastive Continual Learning (2021, methodology): “Despite their simplicity, such methods are known to be surprisingly effective; for ImageNet classification [37], contrastively trained representations closely achieve the fully-supervised performance even without labels [10] and outperform counterparts in the supervised case [24].”
New Insights on Reducing Abrupt Representation Change in Online Continual Learning (2021, methodology): “We use the following loss, denoted SupCon (Khosla et al., 2020), on the incoming data X in . where sim ( a, b ) = exp( a T b τ (cid:107) a (cid:107)(cid:107) b ) computes the exponential cosine similarity between two vectors, with scaling factor τ (Qi et al., 2018; He et al., 2020). while The stream”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, methodology): “Intuitively, SCL aims to tightly cluster embeddings of samples from the same class while pushing those of different classes further apart.”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Many works use a memory bank to store the instance class representation vector [67, 77, 59, 23, 44, 9], while some others explore the usage of in-batch negative samples instead [14, 71, 28, 31].”