Revisiting Neural Scaling Laws in Language and Vision

By Ibrahim M. Alabdulmohsin, Behnam Neyshabur, Xiaohua Zhai. 2022

168 citations recorded by the source corpus.

The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Dataset reinforcement significantly benefits from efficiently reusing the knowledge of a teacher. models have gradually grown larger in multiple domains [1].”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, unknown): “Increasing model size and training tokens typically leads to performance improvements, as known as scaling laws, which have been extensively explored in the language and vision domains (Kaplan et al., 2020; Alabdulmohsin et al., 2022).”
Training Trajectories of Language Models Across Scales (2022, background): “Previous work studies scaling on downstream tasks (Wei et al., 2022; Srivastava et al., 2022), pre-training data (Hernandez et al., 2022), architectures (Tay et al., 2022), biases (Tal et al., 2022), and other domains, such as vision tasks and neural machine translation (Alabdulmohsin et al., 2022).”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “…et al. 2023; Team 2023; Bi et al. 2024] and incredible scalability [Kaplan et al. 2020; Henighan et al. 2020; Hoffmann et al. 2022; Wei et al. 2022; Alabdulmohsin et al. 2022; Chowdhery et al. 2023; Anil et al. 2023], demonstrating a promising path toward general-purpose artificial intelligence…”
Scaling Data-Constrained Language Models (2023, methodology): “1 The metric used to quantify progress is the model’s loss on held-out data, i.e. the ability to predict the underlying data as measured in the model’s cross-entropy [2, 42].”
Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval (2026, unknown): “While effective, this paradigm may not fully leverage the generative power of MLLMs.”