A Constructive Prediction of the Generalization Error Across Scales

By Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit. 2019

278 citations recorded by the source corpus.

The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Neural Language Models (2020, positive): “Very recent work [RRBS19b] studies scaling with both dataset size and model size for a variety of datasets, and ﬁts an ansatz similar to ours.”
LLaMA: Open and Efficient Foundation Language Models (2023, background): “Hestness et al. (2017) and Rosenfeld et al. (2019) studied the impact of scaling on the performance of deep learning models, showing the existence of power laws between the model and dataset sizes and the performance of the system.”
Unified Scaling Laws for Routed Language Models (2022, background): “…that the performance of models on many interesting classes of problems are well understood as power-laws; where a multiplicative increase in model size leads to an additive reduction in the model’s loss [Kaplan et al., 2020, Hernandez et al., 2021, Henighan et al., 2020, Rosenfeld et al., 2019].”
Scaling Laws for Transfer (2021, methodology): “It may also be the case that a modiﬁed version of E.1 of the type proposed in [RRBS19] would be more appropriate.”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “Further, it is recognized that larger datasets require larger architectures to realize benefits, and vice versa [25,45].”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “Performance gains from increasing data, parameters, and training are not only reliable, but empirically predictable (Hestness et al. 2017; Sun et al. 2017; Rosenfeld et al. 2020; Kaplan et al. 2020).”
On the Predictability of Pruning Across Scales (2020, methodology): “(grey) Actual error, (blue) error of our scaling law and (red) the scaling law adapted from Rosenfeld et al. [2020]. (center) Estimated error from our scaling law as width varies for the CIFAR-10 ResNets.”
A Neural Scaling Law from the Dimension of the Data Manifold (2020, background): “5 (for the error rate) [RRBS19] and a smaller ID ≈ 8 [ALMZ19].”
Predictability and Surprise in Large Generative Models (2022, background): “Subsequent work formalized and experimentally validated a quantitative relationship between scale (in terms of both model size and training data size) and model generalization error [66].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “Early work in scaling laws highlighted the importance of studying how variables change with scale since scale is one of the best predictors of model performance (Kaplan et al., 2020; Rosenfeld et al., 2019; Hestness et al., 2017).”
Scaling Laws for Neural Machine Translation (2021, background): “…theoretical understanding of the factors governing the quality and training dynamics of large neural networks has also being developing (Advani et al., 2020; Rosenfeld et al., 2019; Geiger et al., 2020; Ghorbani et al., 2021; Cho, 2020; Hutter, 2021; Bahri et al., 2021; Loureiro et al., 2021).”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, background): “Increasing the model dimensions (width, depth and resolution) has been another successful axis to improve quality [44, 17].”
Data Movement Is All You Need: A Case Study on Optimizing Transformers (2020, methodology): “To help guide training regimes for transformers, recent work has provided empirical recommendations on model size, batch size, and so on (Li et al., 2020; Kaplan et al., 2020; Rosenfeld et al., 2020).”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, background): “…resources (e.g. data, parameter count, compute) scale when optimizing for neural network performance has received significant attention as machine learning models continue to stretch computational limits [Hestness et al., 2017, Rosenfeld et al., 2020, Kaplan et al., 2020, Bahri et al., 2021].”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “More speciﬁcally, in a standard learning setting without transfer, a “power law”-like relationship called a scaling law often holds between data size and generalization errors [35, 53].”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, background): “A primary driver of recent success in machine learning has been the rise of self-supervised learning (SSL) scaled to ever larger models and unlabelled datasets [1, 2, 3, 4, 5, 6, 7, 8].”