Beyond human-level accuracy: computational challenges in deep learning

By Joel Hestness, Newsha Ardalani, G. Diamos. 2019

87 citations recorded by the source corpus.

Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware design, it is difficult to predict DL model changes. However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably. This paper leverages the prior work to project the dataset and model size growth required to advance DL accuracy beyond human-level, to frontier targets defined by machine learning experts. Datasets will need to grow 33--971×, while models will need to grow 6.6--456× to achieve target accuracies. We further characterize and project the computational requirements to train these applications at scale. Our characterization reveals an important segmentation of DL training challenges for recurrent neural networks (RNNs) that contrasts with prior studies of deep convolutional networks. RNNs will have comparatively moderate operational intensities and very large memory footprint requirements. In contrast to emerging accelerator designs, large-scale RNN training characteristics suggest designs with significantly larger memory capacity and on-chip caches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Scaling Laws for Neural Language Models (2020, positive): “More recent work [HNA + 17, HAD19] also investigated scaling between model size and data size; their work is perhaps the closest to ours in the literature 8 .”
A Neural Scaling Law from the Dimension of the Data Manifold (2020, background): “• Scaling holds very accurately across a wide range of N , sometimes spanning many orders of magnitude [HNA + 17, HAD19, KMH + 20].”
Data and Parameter Scaling Laws for Neural Machine Translation (2021, background): “gressive generative models scales like a power law in the amount of training data, compute, and number of model parameters over several orders of magnitude (Hestness et al., 2019; Kaplan et al., 2020; Henighan et al., 2020).”
Accelerating Sparse Deep Neural Networks (2021, methodology): “For example, metadata for the first row of matrix in Figure 1 is [[0, 3], [1, 2]].”