Scaling Language Models: Methods, Analysis & Insights from Training Gopher

By Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young and 70 more. 2021

1,640 citations recorded by the source corpus.

Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.

Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yoshua Bengio
  6. Quoc V. Le
  7. Yejin Choi
  8. Graham Neubig

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Timo Schick
  5. Furu Wei
  6. Hannaneh Hajishirzi
  7. Jianfeng Gao
  8. Danqi Chen

In Scaling & Learning Theory

  1. Luke S. Zettlemoyer
  2. Neil Houlsby
  3. Basil Mustafa
  4. Mostafa Dehghani
  5. Xiaohua Zhai
  6. Lucas Beyer
  7. Ari S. Morcos
  8. Anurag Arnab

Citations in the corpus