BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
By Jacob Devlin, Ming-Wei Chang, Kenton C. T. Lee, Kristina Toutanova. North American Chapter of the Association for Computational Linguistics, 2019
118,136 citations recorded by the source corpus.
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing
View Paper: https://arxiv.org/abs/1810.04805
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Graham Neubig
- Yejin Choi
In General Social Sciences
- Yulia Tsvetkov
- Yonatan Belinkov
- Hinrich Schütze
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
- Noah A. Smith
- Yejin Choi
- Furu Wei
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Furu Wei
- Timo Schick
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Noah A. Smith
- Kyunghyun Cho
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “, 2019] or classification and span prediction [Devlin et al., 2018; Yang et al., 2019]), we found that using a standard encoder-decoder structure achieved good results on both generative and classification tasks.”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “We find that this setting outperforms the originally published BERT BASE results and that removing the NSP loss matches or slightly improves downstream task performance , in contrast to Devlin et al. (2019).”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Although we follow the two-segment data format, XLNet-Large does not use the objective of next sentence prediction [10] as it does not show consistent improvement in our ablation study (see Section 3.4).”
- Scaling Laws for Neural Language Models (2020, background): “Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many specific tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC +…”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Broadly, language modeling refers to approaches for predicting either the next token in a sequence or for predicting masked spans (Devlin et al., 2019; Raffel et al., 2020).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Notable models include BERT (Devlin et al., 2018), GPT-2 (Radford et al.”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “DPR follows a bi-encoder architecture: where d ( z ) is a dense representation of the document produced by a BERT BASE transformer [8], and q ( x ) a representation of the query by another BERT BASE transformer with a different set of parameters.”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Examples of this progression include models that condition on surrounding words (Mikolov et al., 2013a;b), sentences (Kiros et al., 2015; Peters et al., 2018), and paragraphs (Radford et al., 2018; Devlin et al., 2018).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The inverse cloze task was proposed by Lee et al. (2019) to pre-train dense retrievers in an unsupervised way.”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “We base ViT configurations on those used for BERT (Devlin et al., 2019), as summarized in Table 1.”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
- Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “, [47, 14, 48, 4]) enable benefits from exponentially scaling models.”
- End-to-End Object Detection with Transformers (2020, methodology): “Compared to most previous work on direct set prediction, the main features of DETR are the conjunction of the bipartite matching loss and transformers with (non-autoregressive) parallel decoding [29,12,10,8].”
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “We employ the BERT-tokenizer [14] and implement τθ as a transformer [94] to infer a latent code which is mapped into the UNet via cross-attention (Sec.”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “Unsupervised representation learning is highly successful in natural language processing, e.g ., as shown by GPT [50, 51] and BERT [12].”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “The NLP landscape has recently been revolutionized by language models (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020, inter alia ).”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “As LAMA was originally designed to evaluate masked LMs (e.g., Devlin et al., 2019), we filter out examples where the mask token is not the final token, so that all examples can be processed in a left-to-right fashion.”