CLEAR: Contrastive Learning for Sentence Representation

By Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, Hao Ma. 2020

350 citations recorded by the source corpus.

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021).”
Generating Datasets with Pretrained Language Models (2021, methodology): “…differs from that described in Reimers and Gurevych (2019) (and used by all other baselines) in the following respects: • Both Giorgi et al. (2020) and Wu et al. (2020) treat SICK and STSb as supervised tasks, i.e., they use the provided task-specific training sets to perform regular supervised…”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, background): “To minimize the likelihood of incorrect (Utterance, label) pairs, unlike (Chen et al., 2020; Wu et al., 2020; Khosla et al., 2020; Gunel et al., 2021), we fine-tune GPT-2 on a supervised contrastive learning objective.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “Several recent (and concurrent) approaches adopt contrastive objectives (Zhang et al., 2020; Giorgi et al., 2021; Wu et al., 2020; Meng et al., 2021; Carlsson et al., 2021; Kim et al., 2021; Yan et al., 2021) by taking different views—from data augmentation or different copies of models—of the same ”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “The methods vary in how they form these representations – perturbing the input sentence (Wu et al., 2020), or using different model instances (Carlsson et al., 2021).”
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding (2021, background): “Wu et al. (2020) present multiple sentence-level augmentation strategies for contrastive sentence representation learning.”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, background): “Recent works have brought the idea to natural language processing to learn high-quality sentence representation (Giorgi et al., 2020; Wu et al., 2020).”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, methodology): “[Wu et al., 2020b] used approaches such as word deletion, span deletion, token reordering, and synonym substitution for sentence augmentation in their CL method, CLEAR.”
A Survey on Data Augmentation for Text Classification (2021, neutral): “[49] use several simple methods of data augmentation for the contrastive learning objective.”
Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2021, neutral): “, word deletion, reordering, and substitution [26, 47].”
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021, methodology): “CLEAR (Wu et al., 2020) uses the same architecture and objectives as DeCLUTR.”
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup (2021, methodology): “Recent works in NLP adopt deep neural nets as encoders and use unsupervised contrastive learning on sentence representation (Giorgi et al., 2020), text retrieval (Lee et al., 2019), and language model pre-training tasks (Wu et al., 2020).”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, neutral): “2The recent work by Wu et al. (2020) has explored input augmentation techniques for contrastive sentence representation learning.”