Scaling Language Models: Methods, Analysis & Insights from Training Gopher
By Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young and 70 more. 2021
1,640 citations recorded by the source corpus.
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Yejin Choi
- Graham Neubig
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Timo Schick
- Furu Wei
- Hannaneh Hajishirzi
- Jianfeng Gao
- Danqi Chen
In Scaling & Learning Theory
- Luke S. Zettlemoyer
- Neil Houlsby
- Basil Mustafa
- Mostafa Dehghani
- Xiaohua Zhai
- Lucas Beyer
- Ari S. Morcos
- Anurag Arnab
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “The compute and energy cost for training large language models is substantial (Rae et al., 2021; Thoppilan et al., 2022) and rises with increasing model size.”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “To evaluate toxicity degeneration, we adapt the methodology used in Welbl et al. (2021); Rae et al. (2021).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Lazaridou et al. (2022) explore NaturalQuestions in a 15-shot setup using Gopher, augmenting questions with 50 passages retrieved using Google Search.”
- Training language models to follow instructions with human feedback (2022, methodology): “…intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “For the traditional few-shot prompting method used in Brown et al. (2020), it works poorly on tasks that require reasoning abilities, and often does not improve substantially with increasing language model scale (Rae et al., 2021).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Since then, several other hundred-billion scale dense transformer models have been announced, among them are Jurassic-1 [34], Yuan 1.0 [74], PanGu-α [78] and Gopher [53].”
- Improving language models by retrieving from trillions of tokens (2021, methodology): “4 shows the relative improvements in bits-per-byte over our 7B transformer baseline for our Gopher, and .”
- Solving Quantitative Reasoning Problems with Language Models (2022, background): “In particular, large language models have achieved excellent performance across a variety of natural language tasks including common-sense reasoning, question answering, and summarization [1, 2, 3, 4, 5].”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
- Competition-level code generation with AlphaCode (2022, methodology): “Table 7 compares our base 1B model trained on our full GitHub dataset with equivalent models that are pretrained on (1) the Python-only portion of GitHub, (2) the MassiveText generic text dataset (Rae et al., 2021) which also includes a portion of GitHub or (3) not pre-trained at all.”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “, 2022) and exploit (Shoeybi et al., 2019; Brown et al., 2020; Smith et al., 2022; Chowdhery et al., 2022; Rae et al., 2021; Wang et al., 2021; Zeng et al., 2021; Zhang et al., 2022) the benefits of scale.”
- Emergent Abilities of Large Language Models (2022, positive): “…the following general definition of emergence, adapted from Steinhardt (2022) and rooted in a 1972 essay called “More Is Different” by Nobel prize-winning physicist Philip Anderson (Anderson, 1972): Emergence is when quantitative changes in a system result in qualitative changes in behavior.”