Deep Learning Scaling is Predictable, Empirically
By Joel Hestness, Sharan Narang, Newsha Ardalani, G. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, Yanqi Zhou. 2017
1,031 citations recorded by the source corpus.
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Quoc V. Le
- Yejin Choi
- Yoshua Bengio
- Noah A. Smith
In Scaling & Learning Theory
- Greg Yang
- Sam Schoenholz
- Lechao Xiao
- Luke S. Zettlemoyer
- Jascha Narain Sohl-Dickstein
- Jaehoon Lee
- Matthieu Wyart
- Yasaman Bahri
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b].”
- LLaMA: Open and Efficient Foundation Language Models (2023, background): “Hestness et al. (2017) and Rosenfeld et al. (2019) studied the impact of scaling on the performance of deep learning models, showing the existence of power laws between the model and dataset sizes and the performance of the system.”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We study the scalability of CLIP by training a series of eight models spanning almost 2 orders of magnitude of compute and observe that transfer performance is a smoothly predictable function of compute (Hestness et al., 2017; Kaplan et al., 2020).”
- A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Most relevant to our work, Hestness et al. (2017) explored the effect of data size on the generalization error in vision, language, and speech tasks, and observed a strikingly consistent power-law behavior in a large set of experiments.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “While deeper models show great performance improvements, they come bundled with high decoding latency, a significantly larger computational foot-print, and trainability concerns including vanishing/exploding gradients, early divergence, ill-conditioned initial conditions etc. (Hestness et al., 2017).”
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “Larger models trained on larger data sets are partly responsible for these recent successes and, in general, we expect that models trained on more data will continue to yield improvements in predictive performance (Hestness et al., 2017).”
- One Epoch Is All You Need (2019, methodology): “Some works have analyzed the relationship between the performance and the dataset size (Sun et al., 2017; Hestness et al., 2017).”
- Beyond human-level accuracy: computational challenges in deep learning (2019, methodology): “Table 1 lists estimates of α and βд for the di erent modeling tasks as found in the prior work [18].”
- The State of Sparsity in Deep Neural Networks (2019, background): “While model quality has been shown to scale with model and dataset size (Hestness et al., 2017), the resources required to train and deploy large neural networks can be prohibitive.”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “In addition, the empirical observation that a language model’s performance tends to increase as the model is made larger—sometimes predictably (Hestness et al., 2017; Kaplan 1. hf.co/bigscience/bloom et al., 2020; Hoffmann et al., 2022) and sometimes suddenly (Wei et al., 2022)—has led to a trend…”
- Mixture-of-Experts with Expert Choice Routing (2022, background): “The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20].”
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, background): “Their success is partially attributable to their apparent scalability , i.e., the empirical observation that training them on larger datasets produces better performance [25, 17, 35, 46, 34, 18].”
- UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “Performance gains from increasing data, parameters, and training are not only reliable, but empirically predictable (Hestness et al. 2017; Sun et al. 2017; Rosen-feld et al. 2020; Kaplan et al. 2020).”
- On the Predictability of Pruning Across Scales (2020, background): “Hestness et al. (2017) observe a power-law relationship between dataset size and the error of vision and NLP models.”
- Predictability and Surprise in Large Generative Models (2022, background): “This was first noticed by [37] who observed that capabilities such as machine translation and speech recognition increased in a smooth, predictable manner as the size of the model increased.”