SimCSE: Simple Contrastive Learning of Sentence Embeddings

By Tianyu Gao, Xingcheng Yao, Danqi Chen. 2021

4,614 citations recorded by the source corpus.

This paper presents SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework, by using “entailment” pairs as positives and “contradiction” pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman’s correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show—both theoretically and empirically—that contrastive learning objective regularizes pre-trained embeddings’ anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Speciﬁcally, given a sequence of text ( w 1 , ..., w n ), ICT samples a span ( w a , ..., w b ), where 1 We compare our pre-training without using any annotated data to REALM (Guu et al., 2020), SimCSE (Gao et al., 2021) and BM25.”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, methodology): “We use two ways to present sentences: averaged word embeddings using FastText (Joulin et al., 2017) and semantic representations of the sentences obtained from SimCSE (Gao et al., 2021).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “However, there exist approaches [14, 53] that solve regression tasks by training on classification tasks.”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, methodology): “For sentence embedding in retrieval, a pretrained BERT is finetuned on both unsupervised and supervised corpus using SimCSE (Gao et al., 2021).”
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, methodology): “This includes exploring dense retrievers trained directly on the multihop retrieval task, such as those based on ColBERT (Khattab and Zaharia, 2020), or SimCSE (Gao et al., 2021) architectures.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “Inspired by recent progresses in contrastive learning of natural language (Gao et al., 2021; Liu et al., 2021a), DOA uses Dropout (Srivastava et al.”
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “Speciﬁ-cally, we follow the training strategy proposed in SimCSE [47], which using an InfoNCE loss [48] objective to ﬁne-tune our pre-trained RoBERTa model.”
Transformer Memory as a Differentiable Search Index (2022, background): “Dual encoders (Dehghani et al., 2017; Gillick et al., 2018; Gao et al., 2021; Ni et al., 2021; Karpukhin et al., 2020) are a well-established paradigm for retrieval.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The irregular representation space raises the risk of degeneration [37, 55] and often necessitates sophisticated post-adjustment or fine-tuning to improve the sequence representations [16, 30, 32, 60].”
Intent Contrastive Learning for Sequential Recommendation (2022, background): “Contrastive Self-Supervised Learning (SSL) has brought much attentions by different research communities including CV [2, 4, 9, 14, 20] and NLP [7, 8, 29, 50], as well as recommendation [44, 45, 47, 51].”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, background): “[22] Tianyu Gao, Xingcheng Yao, and Danqi Chen.”
Improving Passage Retrieval with Zero-Shot Question Generation (2022, background): “As the training set is not pro-6 Retriever BEIR nDCG@10 Recall@100 Baselines BERT (Devlin et al., 2019) 9.3 20.1 SimCSE (Gao et al., 2021) 27.4 48.1 REALM (Guu et al., 2020) 25”
Vision-Language Pre-Training with Triple Contrastive Learning (2022, methodology): “Different from [14] that uses in-batch negatives, we use the same negative text queue T̃ in Equation 1 instead.”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “In the ﬁeld of NLP, numerous approaches have been proposed to learn better token-level [73] and sentence-level [22, 43] representations using contrastive learning.”
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, methodology): “…Dense retrieval models can be categorized into two groups, passage-level retrievers (Karpukhin et al., 2020; Oguz et al., 2021; Xiong et al., 2021; Gao and Callan, 2022) and token/phrase-level retrievers (Khattab and Za-haria, 2020; Khattab et al., 2020; Gao et al., 2021a; Lee et al., 2021b).”
Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document (2021, methodology): “…similarity between the Input sentence vs. each sentence from the Body , and we only keep the four highest scores (6 features); • SimCSE (Gao et al., 2021): Similarly to SBERT, we compute the cosine similarity be-tween the SimCSE embeddings of the Input sentence against the VeriﬁedStatement ,…”