DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

By John Giorgi, Osvald Nitski, Gary D. Bader, Bo Wang. 2020

545 citations recorded by the source corpus.

Sentence embeddings are an important component of many natural language processing (NLP) systems. Like word embeddings, sentence embeddings are typically learned on large text corpora and then transferred to various downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sentence embeddings require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. In this paper, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Inspired by recent advances in deep metric learning (DML), we carefully design a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. When used to extend the pretraining of transformer-based language models, our approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. Importantly, our experiments suggest that the quality of the learned embeddings scale with both the number of trainable parameters and the amount of unlabelled training data. Our code and pretrained models are publicly available and can be easily adapted to new domains or used to embed unseen text.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “, 2020), which maximizes the agreement between global and local features; 2) DeCLUTR (Giorgi et al., 2021), which takes different spans from the same document as positive pairs; 3) CT (Carlsson et al.”
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup (2021, methodology): “Recent works in NLP adopt deep neural nets as encoders and use unsupervised contrastive learning on sentence representation (Giorgi et al., 2020), text retrieval (Lee et al., 2019), and language model pre-training tasks (Wu et al., 2020).”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “A series of subsequent works have demonstrated that task-agnostic self-supervised contrastive pre-training has been shown to improve language modeling [236], [237], [238].”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021).”
A Cookbook of Self-Supervised Learning (2023, background): “For contrastive learning, positive pairs often come from masking and/or cropping input sequences [Meng et al., 2021, Giorgi et al., 2021].”
SGPT: GPT Sentence Embeddings for Semantic Search (2022, methodology): “We follow recent work [16, 15, 20, 27] and perform supervised contrastive learning with in-batch negatives.”
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, methodology): “We use a ResNet-50 model trained with BYOL [16] on ImageNet [38] as the image encoder and transformer models trained with DeCLUTR (sci-base) [15], BERT (base-uncased) [10], RoBERTa (base) [30], and DistilBERT (base-uncased) [39].”
A Survey on Contrastive Self-supervised Learning (2020, methodology): “ONO [75] RTE, COPA, ReCoRD Discourse ﬁne-grained sentence ordering in text INFOXLM [76] MMLM Learning cross-lingual representations CERT [77] GLUE benchmark Capturing sentence-level semantics DeCLUTR [78] OpenWebText Learning universal sentence representations Table 6: Recent contrastive learning me”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, unknown): “Recently, DeCLUTR (Giorgi et al., 2021) follows the distributional hypothesis and formulates sentence embedding training as a contrastive learning task where span pairs sampled from the same document are treated as positive pairs.”
Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning (2022, background): “Since the success of contrastive learning in CV [2, 14], contrastive learning has been widely applied on NLP [5], graph data mining [17, 34] and recommender systems [28, 37].”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, background): “Recent works have brought the idea to natural language processing to learn high-quality sentence representation (Giorgi et al., 2020; Wu et al., 2020).”
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation (2021, methodology): “Therefore, we initialize our model with an image encoder pre-trained on ImageNet[7] 1k and a pretrained universal sentence encoder, such as Sentence Transformers[33] or DeCLUTR[13].”
CLEAR: Contrastive Learning for Sentence Representation (2020, unknown): “Similar to (Giorgi et al., 2020), for the purpose of grabbing both token-level and sentence-level features, we use a combined loss of MLM objective and CL objective to get the overall loss: where L MLM is calculated through predicting the random-masked tokens in set { s } as described in BERT and…”
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021, methodology): “Moreover, we propose multiple data augmentation strategies for contrastive learning, including adversarial attack (Goodfellow et al., 2014; Kurakin et al., 2016), token shufﬂing, cutoff (Shen et al., 2020) and dropout (Hinton et al., 2012), that effectively transfer the sentence representations to…”