ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

By Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, Weiran Xu. Annual Meeting of the Association for Computational Linguistics, 2021

639 citations recorded by the source corpus.

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/077c713bccd9d2c7fde68d4cbde06ab0f07a6855

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, methodology): “[Yan et al., 2021] not only used lexical transformation and dropout approaches for data augmentation but also perturbed the input by applying Fast Gradient Value (FSV) [Rozsa et al.”
A Survey on Data Augmentation for Text Classification (2021, positive): “For example, Yan et al. [145] and Wu et al. [49] use several simple methods of data augmentation for the contrastive learning objective.”
Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2021, methodology): “In contrastive learning in the language modeling, ConSERT [52] introduces traditional augmentation methods such as cropping and reordering into the sentence augmentation as positive pairs.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “ConSERT [230] uses four ways to generate different views (i.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “…approaches adopt contrastive objectives (Zhang et al., 2020; Giorgi et al., 2021; Wu et al., 2020; Meng et al., 2021; Carlsson et al., 2021; Kim et al., 2021; Yan et al., 2021) by taking different views—from data augmentation or different copies of models—of the same sentence or document.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “Recently many approaches based on supervised learning [190], [191] or self-supervised learning [158], [159], [192]–[194], [194], [195] are proposed.”