A Neural Scaling Law from the Dimension of the Data Manifold

By Utkarsh Sharma, Jared Kaplan, Jared Kaplan. 2020

69 citations recorded by the source corpus.

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \propto N^{-\alpha}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension $d$. This simple theory predicts that the scaling exponents $\alpha \approx 4/d$ for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of $d$ and $\alpha$ by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Transfer (2021, background): “[SK20] attempts to explain the power-law from [KMH + 20] as arising from neural networks performing regression on a data manifold of intrinsic dimension d .”
Scaling Laws for Reward Model Overoptimization (2022, background): “Scaling laws have seen substantial success in machine learning for predicting properties of language models [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021] and has led to better theoretical understanding of language models [Sharma and Kaplan, 2020, Bahri et al., 2021].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Recent work has looked at the topic of scaling laws in deep supervised learning (Kaplan et al., 2020; Sharma and Kaplan, 2020; Henighan et al., 2020).”
Explaining neural scaling laws (2021, methodology): “Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma G. Teacher-student experiment for scaling of loss with model size.”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “Indeed, prior studies [5, 55] observed that α is inversely proportional to the intrinsic dimension of the data (e.g., dimension of the data manifold), which is an equivalent concept of data complexity.”
KAN: Kolmogorov-Arnold Networks (2024, unknown): “We find that an extremely small [17 , 1 , 14] KAN is able to achieve 81 .”