5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

By 知秀柴田, 知秀柴田. 2020

14,205 citations recorded by the source corpus.

Subfields: Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Model Architectures

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Replacing [MASK] with original tokens as in [10] does not solve the problem because original tokens can be only used with a small probability — otherwise Eq.”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Examples of this progression include models that condition on surrounding words (Mikolov et al., 2013a;b), sentences (Kiros et al., 2015; Peters et al., 2018), and paragraphs (Radford et al., 2018; Devlin et al., 2018).”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “Since the BookCorpus is no longer publicly available, we follow Devlin et al. (2019) and crawl http://smashwords.com .”
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, methodology): “The final training objective is a linear combination of the distillation loss Lce with the supervised training loss, in our case the masked language modeling loss Lmlm [Devlin et al., 2018].”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Following the learning procedure in the previous work (Devlin et al., 2018), we treat these two tasks as the problem of sequence labeling which predicts the possibility of each token as the start or end of answer span.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “Other training parameters are kept the same as (Devlin et al., 2018).”
Defending Against Neural Fake News (2019, methodology): “Our smallest model, Grover-Base, has 12 layers and 124 million parameters, on par with GPT and BERT-Base (Radford et al., 2018; Devlin et al., 2018).”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “Actually, the masked language modeling in BERT (Devlin et al., 2018) and the standard language modeling (Bengio et al., 2003; Mikolov et al., 2010) in GPT (Radford et al., 2018) can be viewed as special cases of MASS.”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “We adopt the Transformer model [60] used in BERT [13] as our baseline model due to its great performances on many tasks.”
Cross-lingual Language Model Pretraining (2019, methodology): “Unlike Devlin et al. (2018), a sequence in a mini-batch can contain more than two consecutive sentences, as explained in Section 3.2.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Second, though it is possible to use padding to respect the sentence or other semantic boundaries, in practice it has been standard practice to simply chunk long text into fixed-length segments due to improved efficiency (Peters et al., 2018; Devlin et al., 2018; Al-Rfou et al., 2018).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “, 2015), or contextualized word vectors (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2018) for transfer learning.”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, methodology): “In particular, we first treat the pre-training procedure as a Cloze task as in BERT [1] by masking some percentage of the input tokens at random and then predicting only these masked tokens.”
Parameter-Efficient Transfer Learning for NLP (2019, methodology): “We first evaluate on GLUE.2 For these datasets, we transfer from the pre-trained BERTLARGE model, which contains 24 layers, and a total of 330M parameters, see Devlin et al. (2018) for details.”
Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “…limit, the output of the network at initialization is a draw from a Gaussian process (GP); moreover, the network output remains governed by a GP after exact Bayesian training using squared loss (Neal, 1994; Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2018).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “On ANLI, the supervised baseline we compare to is InfoBERT [68].”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “, 2017) have become ubiquitous in the modern deep learning stack and have seen widespread impact across not only language (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020) but also computer vision (Dosovitskiy et al.”
Measuring Massive Multitask Language Understanding (2020, background): “Until recently, researchers primarily used fine-tuned models on downstream tasks (Devlin et al., 2019).”