Cloze-driven Pretraining of Self-attention Networks

By Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke S. Zettlemoyer, Michael Auli. 2019

210 citations recorded by the source corpus.

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with BERT. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Some recent notable exceptions (Baevski et al., 2019; Liu et al., 2019c; Yang et al., 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “Baevski et al. (2019) demonstrate that increasing data size can result in improved end-task performance.”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “Language model (LM) pre-training has substantially advanced the state of the art across a variety of natural language processing tasks (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “More recently, Baevski et al. (2019) pre-trained a BERT-like model on Common Crawl as preprocessed in Grave et al. (2018).”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Unsupervised pre-training of language models (Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019; Song et al., 2019; Dong et al., 2019; Yang et al., 2019; Joshi et al., 2019; Liu et al., 2019; Lewis et al., 2019; Raffel et al., 2019) has achiev”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “Self-supervised learning of generic neural representations has gathered much recent interest with a large body of work in natural language processing (NLP; Radford et al. 2018; Baevski et al. 2019; Devlin et al. 2019; Raffel et al. 2019), computer vision (Chen et al.”
Effectiveness of self-supervised pre-training for speech recognition (2019, methodology): “…model (Devlin et al., 2018; Liu et al., 2019) with only the masked language modeling task on each set of inputs in the same way as described in (Baevski et al., 2019b), namely by choosing tokens for masking with probability of 0.05, expanding each chosen token to a span of 10 masked tokens…”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “In natural language processing (NLP), unsupervised pretraining of language models (Devlin et al., 2018; Radford et al., 2018; Baevski et al., 2019) improved many tasks such as text classification, phrase structure parsing and machine translation (Edunov et al.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “This may suggest that in some cases, continued pretraining on any additional data is useful, as noted in Baevski et al. (2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Several studies explored the benefits of increasing the corpus volume (Liu et al., 2019b; Conneau et al., 2019; Baevski et al., 2019) and longer training (Liu et al.”
Pre-trained language model representations for language generation (2019, background): “The bi-directional model solves a cloze-style token prediction task at training time (Baevski et al., 2019 N = 12 transformer blocks.”
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019, background): “The ability to learn such representations improves signiﬁcantly with larger training datasets (Baevski et al., 2019; Radford et al., 2019).”
Visualizing and Understanding the Effectiveness of BERT (2019, background): “Language model pre-training has achieved strong performance in many NLP tasks (Peters et al., 2018; Howard and Ruder, 2018a; Radford et al., 2018; Devlin et al., 2018; Baevski et al., 2019; Dong et al., 2019).”
Quantity doesn’t buy quality syntax with neural language models (2019, neutral): “Dra-matically increasing the pre-training corpus for a BERT-like model from 562M words to 18G words only leads to a modest improvement in its natu-ral language inference accuracy, from 81.7% to 82.3% (Baevski et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, background): “[5] proposed a two-tower model that the forward tower operates the left-to-right LM and the backward tower operates the right-to-left LM.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “, 2018), bi-directional (Peters et al., 2018; Baevski et al., 2019), masked (Devlin et al.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “Self-supervised learning in NLP. Pre-training has been very successful in advancing natural language understanding (McCann et al., 2017; Peters et al., 2018; Radford et al., 2018; Baevski et al., 2019; Devlin et al., 2019; Yang et al., 2019; Brown et al., 2020).”
CamemBERT: a Tasty French Language Model (2019, background): “Finally, large pretrained architectures settled the current state of the art showing a small yet important improvement over previous NER-speciﬁc architectures (Devlin et al., 2019; Baevski et al., 2019).”
Provable Meta-Learning of Linear Representations (2020, background): “ranging from deep reinforcement learning (Baevski et al., 2019) to natural language processing (Ando & Zhang, 2005; Liu et al.”