Training Compute-Optimal Large Language Models

By Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark and 12 more. Advances in Neural Information Processing Systems 35, 2022

3,535 citations recorded by the source corpus.

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory

View Paper: https://arxiv.org/abs/2203.15556

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yoshua Bengio
  6. Quoc V. Le
  7. Graham Neubig
  8. Yejin Choi

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Timo Schick
  4. Yejin Choi
  5. Hannaneh Hajishirzi
  6. Furu Wei
  7. Mohit Bansal
  8. Jianfeng Gao

In Scaling & Learning Theory

  1. Tengyu Ma
  2. Behnam Neyshabur
  3. Hanie Sedghi
  4. Ruslan Salakhutdinov
  5. Neil Houlsby
  6. Quoc V. Le
  7. Julien Mairal
  8. Michael W. Mahoney

Citations in the corpus