Efficient Processing of Deep Neural Networks: A Tutorial and Survey

By Vivienne Sze, Yu-hsin Chen, Tien-Ju Yang, Joel S. Emer. 2017

3,699 citations recorded by the source corpus.

Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems. This article aims to provide a comprehensive tutorial and survey about the recent advances toward the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic codesigns, being proposed in academia and industry. The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the tradeoffs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

What is the State of Neural Network Pruning? (2020, background): “These requirements not only increase infrastructure costs, but also make deployment of networks to resource-constrained environments such as mobile phones or smart devices challenging (Han et al., 2015; Sze et al., 2017; Yang et al., 2017).”
Bayesian Compression for Deep Learning (2017, methodology): “As an illustration, consider the convolutional layers of Alexnet, which account for only 4% of the parameters but 91% of the computation [68].”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “We refer the readers to [21] for a comprehensive survey, and summarize the main approaches below.”
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (2019, background): “In practice, DNN layers vary significantly in memory usage and computational cost (Sze et al., 2017), so these heuristics are not optimal in practice.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “Since mobile hardware is limited in memory capacity and has to be energy efficient, specialized DNN computational hardware is frequently proposed [233].”
Gazelle: A Low Latency Framework for Secure Neural Network Inference (2018, background): “For more details on CNNs, we refer the reader to [37].”
Quantizing deep convolutional networks for efficient inference: A whitepaper (2018, methodology): “Faster inference has been achieved by having efficient kernels for computation in reduced precision like GEMMLOWP [7], Intel MKL-DNN [8] , ARM CMSIS [9], Qualcomm SNPE [10], Nvidia TensorRT [11] and custom hardware for fast inference [12], [13] and [14].”
The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations (2021, methodology): “In the medium term, the energy consumption of the inference phase scales with usage, as inference can usually occur millions of times per day for an indefinite amount of time (Sze et al. 2017).”
Provable and Efficient Continual Representation Learning (2022, background): “DNN pruning has a diverse literature with various architectural, algorithmic, and hardware considerations (Sze et al. (2017); Han et al. (2015b)).”
CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability (2021, background): “MAC operations Number of multiply-accumulate operations are an alternative to reporting compute requirements, in a way that is not inherently tied to specific soft- and hardware (Sze et al., 2017).”
On-Demand Deep Model Compression for Mobile Devices: A Usage-Driven Model Selection Framework (2018, methodology): “Compression is a commonly employed technique to trim down the complexity of DNNs, which can be performed by reducing the weight precision, or the number of operations, or both [41].”
Structured Probabilistic Pruning for Convolutional Neural Network Acceleration (2017, background): “rows and columns of weight matrix), so that it can accelerate CNN computation without specific implementation and hardware modification [1, 26].”
FPGA-Based Accelerators of Deep Learning Networks for Learning and Classification: A Review (2019, methodology): “Similar approaches to the DianNao family of techniques are presented in [115] with similar limitations.”
Accelerating Drugs Discovery with Deep Reinforcement Learning: An Early Approach (2018, methodology): “…the applications for DL have grown exponentially, mainly thanks to more powerful computing platforms such as Graphics Processing Units (GPUs) [53] or specific hardware accelerators such as the Google TPU [22], mature software packages and architectures, new algorithmic techniques, and strong…”
Exploiting time-frequency patterns with LSTM-RNNs for low-bitrate audio restoration (2019, neutral): “Thanks to their capability of extracting representative features, CNNs have been widely and successfully applied in image classification [58].”