Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

By Beliz Gunel, Jingfei Du, Alexis Conneau, Ves Stoyanov. 2020

614 citations recorded by the source corpus.

State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. Cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, the SCL loss we propose obtains improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in both the high-data and low-data regimes, and it does not require any specialized architecture, data augmentation of any kind, memory banks, or additional unsupervised data. We also demonstrate that the new objective leads to models that are more robust to different levels of noise in the training data, and can generalize better to related tasks with limited labeled task data.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Online Continual Learning with Contrastive Vision Transformer (2022, background): “Inspired by recent breakthroughs in self-supervised learning [16,11,30,72], we find that the knowledge learned by supervised contrastive learning [31,41,28] reveals greater robustness and transferability.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Concretely, following the framework proposed in [10, 52], SCL consists of three main components.”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, background): “To minimize the likelihood of incorrect (Utterance, label) pairs, unlike (Chen et al., 2020; Wu et al., 2020; Khosla et al., 2020; Gunel et al., 2021), we fine-tune GPT-2 on a supervised contrastive learning objective.”
Continual Few-Shot Learning for Text Classification (2021, background): “Gunel et al. (2021) proposed supervised contrastive learning for better generalizability, where they jointly optimize the cross-entropy loss and supervised contrastive loss that captures the similarity between examples belonging to same class while contrasting with examples from other classes.”
Entailment as Few-Shot Learner (2021, methodology): “…(i) optimization and regularization techniques during the fine-tuning (Howard & Ruder, 2018; Lee et al., 2019; Zhang et al., 2020), (ii) semi-supervised learning to augment training data (Xie et al., 2020), and (iii) supervised contrastive learning as additional objective (Gunel et al., 2021).”
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2020, neutral): “five subsets of supervised data, and the reported numbers, unless zero-shot, are the average of the top three results of the five runs following previous work (Gunel et al., 2020).”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “Its effectiveness of in language is more observed in the fine-tuning stage, for example, in sentence representation [16], dense retrieval [60], and GLUE fine-tuning [19].”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “, 2020), and contrastive learning (Gunel et al., 2021).”
Improving language models fine-tuning with representation consistency targets (2022, positive): “This was shown to improve finetuning process in (Gunel et al., 2020) for few-shot finetuning.”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “Very recently, one work [19] explored contrastive learning to fine-tune language models.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “To this end, Gunel et al. [80] combined the cross-entropy loss with a supervised contrastive learning loss that pushes the words from the same class close and the words from different classes further apart.”
Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation (2021, background): “CrossEntropy (CE) loss is sensitive to adversarial samples and leads to poor results if the inputs differ from the training data even a bit [9].”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “out [301] c) smaller learning rates in early epochs and fine-tuning the model for more number of epochs [303] and d) use of supervised contrastive loss along with cross-entropy loss [304].”
Intent Contrastive Learning for Sequential Recommendation (2022, background): “Contrastive Self-Supervised Learning (SSL) has brought much attentions by different research communities including CV [2, 4, 9, 14, 20] and NLP [7, 8, 29, 50], as well as recommendation [44, 45, 47, 51].”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “With a supervised setup, Gunel et al. (2021) adopt the contrastive objective to ﬁne-tune pre-trained models on benchmark language understanding datasets.”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “The contrastive objectives are learning representations that are invariant to different views of positive pairs; while maximizing the distance between negative pairs (Gunel et al., 2020).”