Explaining neural scaling laws

By Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma. 2021

491 citations recorded by the source corpus.

Significance The population loss of trained deep neural networks has been empirically observed to improve as a power law in a variety of large models and datasets. We investigate the origins behind such “scaling laws” and provide a taxonomy for different scaling regimes. Our findings are based on derivations in linear random feature models—which, in addition to being a simple fruitful model, also describe the wide network limit of deep neural networks. We further formulate and verify aspects of scaling based on smoothness in interpolating a data manifold. We support our theory with empirical results in realistic settings. Our work provides insights into scaling laws and bridges the large gap between theory and experiment in modern deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “Scaling laws have seen substantial success in machine learning for predicting properties of language models [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021] and has led to better theoretical understanding of language models [Sharma and Kaplan, 2020, Bahri et al., 2021].”
Scaling Laws for Neural Machine Translation (2021, background): “…theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing (Advani et al., 2020; Rosenfeld et al., 2019; Geiger et al., 2020; Ghorbani et al., 2021; Cho, 2020; Hutter, 2021; Bahri et al., 2021; Loureiro et al., 2021).”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “While the bulk of work in scaling laws has been empirical, an interesting line of work focuses on theoretically explaining neural scaling laws [51].”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “There have been a lot recent interest in the scaling properties of Transformer models (Kaplan et al., 2020; Hernandez et al., 2021; Bahri et al., 2021; Henighan et al., 2020; Tay et al., 2021b; Abnar et al., 2021).”
Large Language Models (2023, background): “Both the number of parameters and the number of points required to fit this manifold go as the dimension d of the manifold, and this leads to αP = αD = 4/d (the precise coefficient 4 depends on assumptions about smoothness) [8].”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “Bahri et al. [5] addressed power laws under four regimes for model and data size.”
KAN: Kolmogorov-Arnold Networks (2024, unknown): “The origin of NSLs still remains mysterious, but competitive theories include intrinsic dimensionality [52], quantization of tasks [57], resource theory [58], random features [56], compositional sparsity [50], and maximu arity [18].”
Revisiting Neural Scaling Laws in Language and Vision (2022, background): “Under the latter assumptions, it has been argued that the scaling exponent would satisfy c = O(1/d) where d is the intrinsic dimension of the data manifold [2, 21, 32].”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “…investigate the scaling of neural networks to better understand scaling laws and optimize training budget (Hest-ness et al., 2017; Kaplan et al., 2020; Bornschein et al., 2020; Ghorbani et al., 2021; Bahri et al., 2021; Tay et al., 2021; Rae et al., 2021; Hoffmann et al., 2022; Ivgi et al., 2022).”
Scaling Data-Constrained Language Models (2023, background): “…minimize the loss ( L ) subject to a compute resource constraint ( C ) via optimal allocation to N and D as: Currently, there are established best practices for scaling LLMs. Return follows a power-law: loss scales as a power-law with the amount of compute used for training [39, 46, 6, 35, 7, 41].”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, methodology): “For example, scaling laws may be applied to model calibration error as a function of dataset size [1, 51].”