RoBERTa: A Robustly Optimized BERT Pretraining Approach
By Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Michael Lewis, Luke S. Zettlemoyer, Veselin Stoyanov. arXiv.org, 2019
30,530 citations recorded by the source corpus.
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing
View Paper: https://arxiv.org/abs/1907.11692
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Quoc V. Le
- Yoshua Bengio
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
- Furu Wei
- Jianfeng Gao
- Timo Schick
- Hannaneh Hajishirzi
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “To obtain relatively fair comparison with RoBERTa, the experiment in this section is based on full data and reuses the hyper-parameters of RoBERTa, as described in section 3.1.”
- Scaling Laws for Neural Language Models (2020, background): “Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many specific tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC +…”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “We experiment with two choices of the pre-training corpus X : (1) Wikipedia, which is identical to the knowledge corpus Z , and (2) CC-News, our reproduction of the corpus of English news proposed by Liu et al. (2019).”
- How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “Fine-tuning on the benchmark may clue the model into such unintended correlations (Liu et al., 2019a).”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “Fine-tuning of pre-trained language models has become the standard paradigm for building dataset-specific stat-of-the-art systems (Devlin et al., 2019; Liu et al., 2019).”
- StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “…current success of neural network models for language is pretrained representations such as word embeddings (Mikolov et al., 2013; Pennington et al., 2014) and pre-trained language models (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Rad-ford et al., 2019; Liu et al., 2019).”
- Adversarial Training for Large Neural Language Models (2020, methodology): “…on unlabeled text has proven very effective to improve generalization performance for a variety of downstream tasks, as exemplified by BERT (Devlin et al., 2018) and other transformer-based models (Liu et al., 2019c; Radford et al., 2018; Clark et al., 2020; Dong et al., 2019; Bao et al., 2020).”
- Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “For instance, Figure 1 shows that the LSTM model declined by over 35%, while RoBERTa’s generalization performance in fact increases.”
- TTTTTackling WinoGrande Schemas (2020, background): “Since T53B is larger than RoBERTa, it cannot be ruled out that model size alone explains the performance gain.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “Self-supervised Pretraining (MLM) We closely follow the pretraining setup and model from R O BERT A (Liu et al., 2019b) with a few minor exceptions.”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “Big, deep neural language models that have been pre-trained on unlabeled text have proven to be extremely performant when fine-tuned on down-stream Natural Language Processing (NLP) tasks (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019; Raffel et al., 2019).”
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “…the recent success brought by pre-training for NLP applications (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b; Liu et al., 2019), especially for text generation (Radford et al., 2019; Song et al., 2019; Dong et al., 2019; Raffel et al., 2019; Lewis et al.,…”
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “With the rise of pretrained language models such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and ”
- PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “…are easily fooled. has been made in Natural Language Processing through a shift towards large-scale pretrained representations from unlabeled text (Radford et al. 2018; Devlin et al. 2019; Liu et al. 2019), the bulk of the success of this paradigm has been on core abstract tasks and domains.”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “For the second round, we used a more powerful RoBERTa model (Liu et al., 2019b) trained on SNLI, MNLI, an NLI-version2 of FEVER (Thorne et al.”
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “We use the same pre-training data as Liu et al. (2019), consisting of 160Gb of news, books, stories, and web text.”
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, methodology): “For the sake of comparison, the RoBERTa model [Liu et al., 2019] required 1 day of training on 1024 32GB V100.”