Measuring the Algorithmic Efficiency of Neural Networks

By Danny Hernandez, Tom B. Brown. 2020

116 citations recorded by the source corpus.

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Evaluation & Benchmarks

Citations in the corpus

Scaling Laws for Transfer (2021, background): “Three factors drive the advance of AI: algorithmic innovation [HB20], compute [AH18], and data.”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, methodology): “In addition to incorporating an efficiency measure as part of leader boards for various ML tasks, data [97], models7, training algorithms [98], environmental impact must also be considered and adopted by AI system hardware developers.”
Chasing Carbon: The Elusive Environmental Footprint of Computing (2020, background): “number of parallel training nodes and the AI-training carbon footprint [75], [78].”
The Computational Limits of Deep Learning (2020, background): “Efforts to decrease the amount of computation have shown considerable promise, doubling in efficiency every year or so [50].”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, methodology): “CINIC-10 Figure 5: Communication Efficiency (ResNet-56) To compare the computational demand on training, we count the number of FLOPs (floating-point operations) performed on edge using prior methods [62, 63].”