Learning Curve Theory
By Marcus Hutter. 2021
96 citations recorded by the source corpus.
Recently a number of empirical"universal"scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute. In this work we focus on scaling w.r.t. data size $n$. Theoretical understanding of this phenomenon is largely lacking, except in finite-dimensional models for which error typically decreases with $n^{-1/2}$ or $n^{-1}$, where $n$ is the sample size. We develop and theoretically analyse the simplest possible (toy) model that can exhibit $n^{-\beta}$ learning curves for arbitrary power $\beta>0$, and determine whether power laws are universal or depend on the data distribution.
Subfields: Artificial Intelligence, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Orhan Firat
- Ankur Bapna
- Felix Stahlberg
- Maxim Krikun
- Colin Cherry
- Graham Neubig
- Kyunghyun Cho
- Xavier García
In Statistical Learning Theory
Citations in the corpus
- Scaling Laws for Neural Machine Translation (2021, background): “…theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing (Advani et al., 2020; Rosenfeld et al., 2019; Geiger et al., 2020; Ghorbani et al., 2021; Cho, 2020; Hutter, 2021; Bahri et al., 2021; Loureiro et al., 2021).”
- Scaling Scaling Laws with Board Games (2021, background): “Hutter [13] is the most recent contribution in the area, and its introduction provides an exhaustive overview of prior work.”
- Revisiting Neural Scaling Laws in Language and Vision (2022, background): “While theoretical arguments alone seldom predict scaling law parameters in modern neural architectures [2, 21, 32], it has been observed that the benefit of scale could be predicted empirically [3, 4, 9, 17, 18, 20, 22, 23, 28, 30, 31].”
- Explaining neural scaling laws (2021, unknown): “All proofs appear in the supplement.”