The Computational Limits of Deep Learning

By Neil C. Thompson, Kristjan H. Greenewald, Keeheon Lee, Gabriel F. Manso. 2020

707 citations recorded by the source corpus.

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image recognition, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article reports on the computational demands of Deep Learning applications in five prominent application areas and shows that progress in all five is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

Measuring the Carbon Intensity of AI in Cloud Instances (2022, background): “The most accurate and popular AI models today are typically (deep) neural networks, which are most performant on specialized, highly parallelized, and often energy-intensive hardware [43].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “However, this approach is impractical in case of frequent updates because training on the entire augmented dataset would take too long, increasing the energy consumption and carbon footprint of machine learning models [51, 58, 61].”
Flow Matching for Generative Modeling (2022, positive): “Furthermore, the energy demand for training large deep learning models is increasing at a rapid pace (Amodei et al., 2018; Thompson et al., 2020), focusing on methods that are able to train using less gradient updates / image throughput can lead to significant time and energy savings.”
The hardware lottery (2020, background): “For many subﬁelds, we are now in a regime where the rate of return for additional parameters is decreasing (Thompson et al., 2020a; Brown et al., 2020).”
Avalanche: an End-to-End Library for Continual Learning (2021, background): “Efficiency and Scalability Computational and memory requirements in machine learning have grown significantly throughout the last two decades [51].”
Twelve Scientific Challenges for 6G: Rethinking the Foundations of Communications Theory (2022, background): “Such estimation can be used to determine the energy requirements of processing hungry applications, such as deep learning (DL) [64].”
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model (2022, background): “Other studies have pursued a broader analysis of trends in terms of the energy requirements and CO2 emissions of ML models in general (Thompson et al., 2020; Wu et al., 2021; Patterson et al., 2022).”
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink (2022, background): “This confusion led them to believe Evolved Transformer used more than 2 million GPU hours to train, cost millions of dollars, and its emissions were five times the lifetime of a car (284,019 kg) [6,7].”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “Over the years, the size of language models have grown exponentially (Amodei et al., 2018; Thompson et al., 2020; Bender et al., 2021).”
Parameter Prediction for Unseen Deep Architectures (2021, background): “[13] Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso.”
Compute Trends Across Three Eras of Machine Learning (2022, background): “This has raised interests in projecting future compute requirements and hardware progress [15, 16], and hints at the growing importance of gathering compute data and investigating trends.”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “The amount of computational power required to run top-performing deep learning models has increased at a rate of 10× per year from 2012 to 2019 [23], [24].”