RoBERTa: A Robustly Optimized BERT Pretraining Approach

By Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Michael Lewis, Luke S. Zettlemoyer, Veselin Stoyanov. arXiv.org, 2019

30,530 citations recorded by the source corpus.

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

View Paper: https://arxiv.org/abs/1907.11692

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Quoc V. Le
  8. Yoshua Bengio

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Mohit Bansal
  5. Furu Wei
  6. Jianfeng Gao
  7. Timo Schick
  8. Hannaneh Hajishirzi

In Natural Language Processing

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Yejin Choi
  4. Graham Neubig
  5. Jianfeng Gao
  6. Percy Liang
  7. Noah A. Smith
  8. Hannaneh Hajishirzi

Citations in the corpus