Training Compute-Optimal Large Language Models
By Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark and 12 more. Advances in Neural Information Processing Systems 35, 2022
3,535 citations recorded by the source corpus.
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory
View Paper: https://arxiv.org/abs/2203.15556
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Graham Neubig
- Yejin Choi
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Timo Schick
- Yejin Choi
- Hannaneh Hajishirzi
- Furu Wei
- Mohit Bansal
- Jianfeng Gao
In Scaling & Learning Theory
- Tengyu Ma
- Behnam Neyshabur
- Hanie Sedghi
- Ruslan Salakhutdinov
- Neil Houlsby
- Quoc V. Le
- Julien Mairal
- Michael W. Mahoney
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, 2021), Chinchilla (Hoffmann et al., 2022), Megatron–Turing NLG (Smith et al.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Open-source releases like BLOOM (Scao et al., 2022) and Falcon (Penedo et al., 2023) have risen to challenge their closed-source counterparts like GPT-3 and Chinchilla.”
- Unified Scaling Laws for Routed Language Models (2022, background): “Prior work has established the notion of an optimal relationship between N and training-token count5 (Kaplan et al., 2020; Hoffmann et al., 2022).”
- Solving Quantitative Reasoning Problems with Language Models (2022, methodology): “, 2015; Hosseini et al., 2014). Prompting language models using scratchpad Nye et al. (2021) or chain-of-thought Wei et al.”
- Emergent Abilities of Large Language Models (2022, methodology): “For example, Chinchilla (Hoffmann et al., 2022) has one-fourth as many parameters as Gopher (Rae et al., 2021) but uses similar training compute; and sparse mixture-of-expert models have more parameters per training/inference compute than dense models (Fedus et al., 2021; Du et al., 2021).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “…Jurassic-113 either uses some form of deduplication (Rae et al., 2022; Askell et al., 2021; Zeng et al., 2021; Sun et al., 2021; Smith et al., 2022; Hoffmann et al., 2022; Chowdhery et al., 2022) or does not discuss the training data in sufficient detail to determine what was done (Kim et al.,…”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “For small-scale and ablation studies (first half of Section 4.2; Section 4.3), we train models to optimality according to the scaling laws of Hoffmann et al. (2022): on 27B and 60B to-kens respectively for our 1B and 3B parameters models.”
- Augmented Language Models: a Survey (2023, background): “Although a recent line of work yielded smaller LMs that retain some capabilities from their largest counterpart (Hoffmann et al., 2022), the size and need for data of LLMs can be impractical for training but also maintenance: continual learning for large models remains an open research question (Sci”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “, 2022), Chinchilla (Hoffmann et al., 2022), and ChatGPT among others (Brown et al.”
- Galactica: A Large Language Model for Science (2022, methodology): “Hoffmann et al. (2022) presented new analysis taking into account the optimal amount of data, and suggested that existing language models were undertrained: "Chinchilla scaling laws".”
- Scaling Instruction-Finetuned Language Models (2022, background): “As a more general comparison, our work is among a growing research direction that aims to improve language models without massively scaling the amount of compute (Hoffmann et al., 2022; Padmakumar et al., 2022).”
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022, methodology): “While answer-only prompting has been used as the standard in many prior work (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Srivastava et al., 2022), it typically underestimates language model performance on challenging tasks, such as those that require multiple reasoning steps.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “Moreover, the Chinchilla scaling laws (Hoffmann et al., 2022) indicate that larger models would benefit more from training on more tokens.”
- Llemma: An Open Language Model For Mathematics (2023, background): “Recent progress in large language models involves two connected threads: the increasing scale of models and data (Hoffmann et al., Language models for mathematics.”
- A Review of Sparse Expert Models in Deep Learning (2022, background): “But in light of the recent scaling results from Hoff-mann et al. (2022) which recommends more tokens to train compute-optimal models (Chinchilla was a 70B parameter model trained on 1.4T tokens), future work might revisit this analysis.”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “For Winogender and Winobias, we follow Brown et al. (2020); Hoffmann et al. (2022); Rae et al. (2021) and select an option by comparing LM likelihoods given a zero-shot dialogue prompt.”