Improving Language Understanding by Generative Pre-Training

By Alec Radford, Karthik Narasimhan. 2018

15,103 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “, 2014) and language model pretraining (Dai and Le, 2015; Ramachandran et al., 2017; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Concurrent with our work, Radford et al. (2018) trained a large forward Transformer LM and ﬁne tuned it for a variety of NLP tasks.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “Unsupervised representation learning has been highly successful in the domain of natural language processing [7, 22, 27, 28, 10].”
Scaling Laws for Neural Language Models (2020, background): “Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many speciﬁc tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC +…”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “The Generative Pretrained Transformer (GPT) series, decoder-only models, (Radford et al., 2018; Raffel et al., 2020) set state-of-the-art language modeling performance.”
LLaMA: Open and Efficient Foundation Language Models (2023, background): “More recently, transformer networks, based on self-attention, have led to important improvements, especially for capturing long range dependencies (Vaswani et al., 2017; Radford et al., 2018; Dai et al., 2019).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “A single, pre-trained language model has been shown to achieve strong performance on various classiﬁcation tasks in the GLUE benchmarks [54, 55] after ﬁne-tuning [43, 8].”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “The solutions, based on autoregressive language modeling in GPT [47, 48, 4] and masked autoencoding in BERT [14], are conceptually simple: they remove a portion of the data and learn to predict the removed content.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “Unsupervised representation learning is highly successful in natural language processing, e.g ., as shown by GPT [50, 51] and BERT [12].”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “…with Language Models In the past few years, it has become increasingly common to pre-train a language model using an unsupervised objective on a large, unstructured text corpus before ﬁne-tuning it on a downstream task of interest (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “With the rise of pretrained language models such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and…”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “As the dataset was constructed to be adversarial to BERT, it is not surprising that it performs the worst of three models despite generally outper-forming GPT on most other benchmarks.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “Documents are tokenized with the same byte-pair encoding as GPT-2 (Radford et al., 2019).”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Motivated by the recent success in self-supervised language representation learning (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Song et al., 2019; Zhang et al., 2019; Yang et al., 2019), we propose several strategies that pre-train the encoder and…”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “…the shift from pre-training word embeddings, whether standard (Mikolov et al., 2013; Pennington et al., 2014) or contextualized (McCann et al., 2017; Peters et al., 2018), to full-network pre-training followed by task-specific fine-tuning (Dai & Le, 2015; Radford et al., 2018; Devlin et al., 2019).”
Fine-Tuning Language Models from Human Preferences (2019, methodology): “…model r : X × Y → R using the loss x,y i ) (1) Since the reward model needs to understand language, we initialize it as a random linear function of the ﬁnal embedding output of the language model policy ρ following Radford et al. (2018) (see section 4.2 for why we initialize from ρ rather than π ).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “However, recent works leveraging transformers for language modeling such as GPT (Radford et al., 2018), BERT (Devlin et al.”