BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

By Jacob Devlin, Ming-Wei Chang, Kenton C. T. Lee, Kristina Toutanova. North American Chapter of the Association for Computational Linguistics, 2019

118,136 citations recorded by the source corpus.

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing

View Paper: https://arxiv.org/abs/1810.04805

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Luke S. Zettlemoyer
  5. Yoshua Bengio
  6. Quoc V. Le
  7. Graham Neubig
  8. Yejin Choi

In General Social Sciences

  1. Yulia Tsvetkov
  2. Yonatan Belinkov
  3. Hinrich Schütze
  4. Ruslan Salakhutdinov
  5. Luke S. Zettlemoyer
  6. Noah A. Smith
  7. Yejin Choi
  8. Furu Wei

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Mohit Bansal
  5. Jianfeng Gao
  6. Furu Wei
  7. Timo Schick
  8. Noah A. Smith

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Jianfeng Gao
  5. Yejin Choi
  6. Percy Liang
  7. Noah A. Smith
  8. Kyunghyun Cho

Citations in the corpus