COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

By Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul N. Bennett, Jiawei Han, Xia Song. 2021

227 citations recorded by the source corpus.

We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “A similar approach has been recently adopted in language representations (Wu et al., 2020; Meng et al., 2021) by applying augmentation techniques such as word deletion, reordering, and substitution.”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “former using the same hidden size as the discriminator (Meng et al., 2021).”
Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2021, unknown): “, word deletion, reordering, and substitution [26, 47].”
Text Transformations in Contrastive Self-Supervised Learning: A Review (2022, methodology): “Meng et al. [Meng et al., 2021] used CL as part of their model for language model pretraining.”
A Cookbook of Self-Supervised Learning (2023, background): “For instance, Feichtenhofer et al. [2021] have generalized SimCLR, MoCo, SwAV and BYOL to space-time video data.”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, background): “Data augmentation methods (Chen et al., 2020; Huang et al., 2022; Lee et al., 2021; Meng et al., 2021b; Miyato et al., 2017; Xie et al., 2020) aim to create similar samples to the existing ones so that the enlarged training set can benefit model generalization.”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021).”
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators (2022, methodology): “We disable dropout in the generator following Meng et al. (2021).”
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021, unknown): “The effectiveness of RTD is also verified by follow-up works, including CoCo-LM [10], XLM-E [11], and SmallBenchNLP [12].”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, unknown): “We also point to very recent work that explores text augmentation in a different context (Wu et al., 2020; Meng et al., 2021).”