Scaling Laws for Neural Language Models
By Jared Kaplan, Jared Kaplan, Sam McCandlish, T. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu and 1 more. arXiv.org, 2020
8,567 citations recorded by the source corpus.
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory
View Paper: https://arxiv.org/abs/2001.08361
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Yejin Choi
- Graham Neubig
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Jianfeng Gao
- Hannaneh Hajishirzi
- Noah A. Smith
- Mohit Bansal
- Furu Wei
In Scaling & Learning Theory
- Tengyu Ma
- Greg Yang
- Behnam Neyshabur
- Lechao Xiao
- Sam Schoenholz
- Hanie Sedghi
- Luke S. Zettlemoyer
- Jaehoon Lee
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, critical): “Our approach leads to considerably different results than that of Kaplan et al. (2020).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “These few-shot properties first appeared when scaling models to a sufficient size (Kaplan et al., 2020), resulting in a line of work that focuses on further scaling these models (Chowdhery et al., 2022; Rae et al., 2021).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “Scaling up language models has been shown to confer benefits such as improved performance and sample efficiency (Kaplan et al., 2020), but chain-of-thought reasoning is emergent in the sense that its success cannot be predicted only by extrapolating the performance of small scale models, as chain of”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “In concurrent work, Kaplan et al. (2020) study the impact of model size on the training efficiency of Transformer language models.”
- Unified Scaling Laws for Routed Language Models (2022, methodology): “Many sparsely-activated networks have this property, but here we exclusively study the layout based on Sparse Mixtures of Experts [Shazeer et al., 2017] where multiple sub-components of a deep neural network (i.e., several layers) are independently converted to routed equivalents and jointly trained”
- LaMDA: Language Models for Dialog Applications (2022, methodology): “Similar to general language models [13], Adiwardana et al.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Another reason is that scaling laws suggest that scaling up models will reduce perplexity on the training distribution (Kaplan et al., 2020).”
- Scaling Laws for Transfer (2021, methodology): “5) We didn’t find a good closed-form model for from-scratch results as was seen in [KMH + 20], though we believe more careful tuning could have produced such a model in line with their results.”
- Measuring Massive Multitask Language Understanding (2020, background): “Current understanding indicates that a 10 × increase in model size must be accompanied by an approximate 5 × increase in data (Kaplan et al., 2020).”
- Competition-level code generation with AlphaCode (2022, methodology): “Following Kaplan et al. (2020), we adjusted the amount of training for other model sizes such that larger models are trained more and smaller models are trained less to optimize the use of compute.”
- Training Verifiers to Solve Math Word Problems (2021, background): “Kaplan et al. (2020) describe the consistent benefits of increasing model size, characterizing scaling trends that hold across many orders of magnitude.”