Language Models are Unsupervised Multitask Learners

By Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever. 2019

29,506 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Early results on transfer learning for NLP leveraged recurrent neural networks [Peters et al., 2018; Howard and Ruder, 2018], but it has recently become more common to use models based on the “Transformer” architecture [Vaswani et al., 2017].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “• O PEN W EB T EXT (Gokaslan and Cohen, 2019), an open-source recreation of the WebText cor-4 pus described in Radford et al. (2019).”
Scaling Laws for Neural Language Models (2020, methodology): “…learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many speciﬁc tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC + 19].”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “, 2018), GPT-2 (Radford et al., 2019), MegatronLM (Shoeybi et al.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “GPT-2 [44] later showed that a single, left-to-right, pre-trained language model could achieve strong performance across both discriminative and generative tasks.”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “Recently, several works have shown that factual information can be extracted from large scale language models trained on vast quantities of data (Radford et al., 2019; Petroni et al., 2019; Jiang et al., 2019; Talmor et al., 2019).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “While it was initially unclear how large amounts of knowledge could be injected into the model, GPT-2 (Radford et al., 2019) hinted at the possibility of directly generating answers without using any given context via sequence-to-sequence.”
Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Providing language models with natural language descriptions of tasks, as proposed by Radford et al. (2019) has led to signiﬁcant developments in few-shot learning.”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…models are often pre-trained on large corpora and then ﬁne-tuned for the task at hand: BERT (Devlin et al., 2019) uses a denoising self-supervised pre-training task, while the GPT line of work uses language modeling as its pre-training task (Radford et al., 2018; 2019; Brown et al., 2020).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “The text encoder is a Transformer (Vaswani et al., 2017) with the architecture modifications described in Radford et al. (2019).”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “The solutions, based on autoregressive language modeling in GPT [40, 41, 4] and masked autoencoding in BERT [14], are conceptually simple: they remove a portion of the data and learn to predict the removed content.”
End-to-End Object Detection with Transformers (2020, neutral): “replacing RNNs in many problems in natural language processing, speech processing and computer vision [7,26,44,33,30].”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “Unsupervised representation learning is highly successful in natural language processing, e.g ., as shown by GPT [50, 51] and BERT [12].”
Training language models to follow instructions with human feedback (2022, methodology): “…intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “Self-supervised language models of increasing scale learn to complete some tasks zero-shot [31] or with few-shot prompts [6, 25, 11].”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “To this end, we apply our approach not just to GPT-J, but also to four smaller models from the GPT-2 family (Radford et al., 2019), with 124M, 355M, 775M and 1.6B parameters, respectively.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Previous work uses textual preﬁxes to explicitly deﬁne the task associated with each input instance (Raffel et al., 2020; Radford et al., 2019b); often such attempts to build a single model for multiple NLP tasks underperform the standard pre-training plus ﬁne-tuning setup (a model per task)…”
Experience Grounds Language (2020, methodology): “For example, in the LAM-BADA dataset (Paperno et al., 2016) (designed to capture human intuition), GPT2 (Radford et al., 2019) (1.5B), Megatron-LM (Shoeybi et al., 2019) (8.3B), and TuringNLG (Rosset, 2020) (17B) perform within a few points of each other and very far from perfect ( 68%).”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “In this section, we evaluate popular pretrained language models such as BERT (Devlin et al., 2019), R O BERT A (Liu et al., 2019), XLN ET (Yang et al., 2019) and GPT2 (Radford et al., 2019) on Stere-oSet.”