CERT: Contrastive Self-supervised Learning for Language Understanding

By Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, Pengtao Xie. 2020

379 citations recorded by the source corpus.

Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In the context of natural language processing, Fang et al. (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natural language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “…have been proposed to change the reference label distributions through label smoothing (Szegedy et al., 2016; M¨uller et al., 2019), Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), knowledge distillation (Hinton et al., 2015) or self-training (Yalniz et al., 2019; Xie et al., 2020).”
CLEAR: Contrastive Learning for Sentence Representation (2020, positive): “Our model differs from CERT in adopting an encoder-only structure, which decreases noise brought by the de-coder.”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “In the context of natural language processing, Fang et al. (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation.”
NewsEmbed: Modeling News through Pre-trained Document Representations (2021, background): “Fang and Xie [22] use back-translation as data augmentation approach that translate a sentence into a target language and back to the source language.”
A Survey on Data Augmentation for Text Classification (2021, neutral): “[99]** Translation models from Britz et al.”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, methodology): “Back-translation n-pairs loss (based on momentum CL) GLUE [Fang et al., 2020]”
Text and Code Embeddings by Contrastive Pre-Training (2022, background): “…training pairs with augmented samples, a diverse set of text augmentation operations have been explored, including lexicon-based distortion (Wei & Zou, 2019), synonym replacement (Kobayashi, 2018), back-translation (Fang & Xie, 2020), cut-off (Shen et al., 2020) and dropout (Gao et al., 2021).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “continual pretraining using contrastive pretraining objectives improves the performance of T-PTLMs in GLUE tasks [157] and multilingual T-PTLMs in Mickey Probe [160].”
Contrastive Code Representation Learning (2020, methodology): “The InfoNCE pre-training objective has been applied to natural language [54, 16], but is outperformed by baselines or requires auxiliary supervised models for data augmentation [20].”
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021, methodology): “CERT (Fang and Xie, 2020) adopts a similar structure as MoCo (He et al., 2020) and uses back-translation for data augmentation.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “Only recently has CL been used for training language models with self-supervision (Fang et al., 2020), learning sentence representations (Gao et al., 2021), and improving document clustering (Zhang et al., 2021a).”