Language Models are Few-Shot Learners
By Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and 22 more. Neural Information Processing Systems, 2020
61,005 citations recorded by the source corpus.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning
View Paper: https://arxiv.org/abs/2005.14165
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
- Sergey Levine
- Yejin Choi
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Mohit Bansal
- Timo Schick
- Jianfeng Gao
- Noah A. Smith
- Furu Wei
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Yejin Choi
- Denny Zhou
- Timo Schick
- Hinrich Schütze
- Swaroop Mishra
- Mohit Bansal
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, positive): “We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “GPT-3 (Brown et al., 2020) demonstrated that extremely large autoregressive language models (LMs) can be used for few-shot predictions, where the model is only given a natural language task description and (optionally) a handful of exemplars demonstrating how the task should be completed.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Following the scaling laws of Kaplan et al. (2020), several Large Language Models with more than 100B parameters have been proposed, from GPT-3 (Brown et al., 2020) to Gopher (Rae et al., 2022) or specialized models, e.g. Galactica, for science(Taylor et al., 2022).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “We find that Atlas is able to perform significantly better than random in zero-shot, and in conjunction with de-biased inference, achieves zero-shot scores that exceed 5-shot results reported with GPT3 in the literature (47.1% vs 43.9”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…models are often pre-trained on large corpora and then fine-tuned for the task at hand: BERT (Devlin et al., 2019) uses a denoising self-supervised pre-training task, while the GPT line of work uses language modeling as its pre-training task (Radford et al., 2018; 2019; Brown et al., 2020).”
- Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “, [40, 14, 41, 4]) enable benefits from exponentially scaling models.”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “To this end, we generated 512 “artistic” captions using GPT-3 [4] by prompting it with captions for existing artwork (both real and AI generated).”
- Training language models to follow instructions with human feedback (2022, methodology): “…intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “Self-supervised language models of increasing scale learn to complete some tasks zero-shot [31] or with few-shot prompts [6, 25, 11].”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, methodology): “In contrast, for decision making tasks that potentially involve a large number of actions (Figure 1(2)), thoughts only need to 1We show some GPT-3 (Brown et al., 2020) results in Appendix A.1, which outperforms PaLM-540B. appear sparsely in the most relevant positions of a trajectory, so we let the ”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Large language models achieve impressive zero and few-shot results on a variety of natural language processing tasks (Brown et al., 2020; Chowdhery et al., 2022, i.a.).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “, [9]), the quality of unfiltered Common Crawl data is lower than that of curated datasets and steps should be taken to increase the average quality of data selected from Common Crawl for LM pretraining.”
- LaMDA: Language Models for Dialog Applications (2022, methodology): “Language model pre-training is an increasingly promising research approach in NLP [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].”