Scaling Laws for Autoregressive Generative Modeling

By T. Henighan, Jared Kaplan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal and 10 more. arXiv.org, 2020

639 citations recorded by the source corpus.

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Probabilistic & Bayesian Methods

View Paper: https://arxiv.org/abs/2010.14701

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Probabilistic & Bayesian Methods

Citations in the corpus

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “This paradigm incentivizes the use of ever larger corpora (Kaplan et al., 2020; Henighan et al., 2020), with the biggest models now training on a substantial fraction of the publicly-available internet (Raffel et al.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps [39, 32].”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, methodology): “Our GPT-2 models tokenizes numbers so that one digit is processed at a time (Henighan et al., 2020).”
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (2021, background): “When increasing model and compute budget scale in addition, scaling laws suggest further increase in generalization and transfer performance if not bottlenecked by the data scale [5, 6, 7, 8].”
High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “Our results back the ﬁnding that autoregressive models increase performance with higher data amount [8, 14] and indicate that this is also true for stronger data augmentation, i.e . a virtual increase of data.”
Improved Denoising Diffusion Probabilistic Models (2021, background): “This could be caused by a variety of factors, such as 1) an unexpectedly high irreducible loss (Henighan et al., 2020) for this type of diffusion model, or 2) the model over-ﬁtting to the training distribution.”
Scaling Laws for Reward Model Overoptimization (2022, background): “Scaling laws have seen substantial success in machine learning for predicting properties of language models [Kaplan et al., 2020, Henighan et al., 2020, Hernandez et al., 2021] and has led to better theoretical understanding of language models [Sharma and Kaplan, 2020, Bahri et al., 2021].”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “It is well established that there are regular and predictable patterns in the behavior of trained language models as they scale (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Mikami et al., 2021; Pu et al., 2021; Sharma & Kaplan, 2020; Ghorbani et al., 2021), but prior work…”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “Several other investigators have also pre-trained or fine-tuned neural models on synthetic data consisting of mathematical problems [3], [21], [22], although these approaches have generally used very large numbers of training examples.”
Predictability and Surprise in Large Generative Models (2022, background): “) [35], audition [21], transfer from text to programming [36], few-shot adaptation of vision models [54], and more.”
Scaling Laws for Acoustic Models (2021, methodology): “Both [1] and [2] showed that for a broad class of models, training to convergence is a computationally inefficient way to maximize model performance.”
Scalable Diffusion Models with Transformers (2022, background): “They have shown remarkable scaling properties under increasing model size, training compute and data in the language domain [26], as generic autoregressive models [17] and as ViTs [63].”