Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
By By Lei Deng, Guoqi Li, Song Han, Luping P. Shi, Yuan Xie. Proceedings of the IEEE, 2020
940 citations recorded by the source corpus.
Domain-specific hardware is becoming a promising topic in the backdrop of improvement slow down for general-purpose processors due to the foreseeable end of Moore’s Law. Machine learning, especially deep neural networks (DNNs), has become the most dazzling domain witnessing successful applications in a wide spectrum of artificial intelligence (AI) tasks. The incomparable accuracy of DNNs is achieved by paying the cost of hungry memory consumption and high computational complexity, which greatly impedes their deployment in embedded systems. Therefore, the DNN compression concept was naturally proposed and widely used for memory saving and compute acceleration. In the past few years, a tremendous number of compression techniques have sprung up to pursue a satisfactory tradeoff between processing efficiency and application accuracy. Recently, this wave has spread to the design of neural network accelerators for gaining extremely high performance. However, the amount of related works is incredibly huge and the reported approaches are quite divergent. This research chaos motivates us to provide a comprehensive survey on the recent advances toward the goal of efficient compression and execution of DNNs without significantly compromising accuracy, involving both the high-level algorithms and their applications in hardware design. In this article, we review the mainstream compression approaches such as compact model, tensor decomposition, data quantization, and network sparsification. We explain their compression principles, evaluation metrics, sensitivity analysis, and joint-way use. Then, we answer the question of how to leverage these methods in the design of neural network accelerators and present the state-of-the-art hardware architectures. In the end, we discuss several existing issues such as fair comparison, testing workloads, automatic compression, influence on security, and framework/hardware-level support, and give promising topics in this field and the possible challenges as well. This article attempts to enable readers to quickly build up a big picture of neural network compression and acceleration, clearly evaluate various methods, and confidently get started in the right way.
Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Hardware and Architecture
View Paper: https://www.semanticscholar.org/paper/e70d609ce18cd61799b087bf3a5e14c1ce70a41a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mathematics
In Computer Vision and Pattern Recognition
In Hardware and Architecture
Citations in the corpus
- Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, neutral): “[26], [46], [47], [48], [49], [50] X X X X X”
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, background): “…often overparameterized and can be pruned significantly with minimal impact on performance (Han et al., 2016; Wen et al., 2016; Liu et al., 2017; Luo et al., 2017; Cai et al., 2019; Deng et al., 2020; Hou et al., 2020; Wang et al., 2020; Lagunas et al., 2021; Xia et al., 2022; Kurtic et al., 2023).”
- The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century (2024, background): “This is where hardware acceleration steps in, acting as a powerful engine that fuels AI-powered diagnostics [111].”
- Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G (2021, methodology): “compression and hardware acceleration for neural networks can be found in [306].”
- Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, neutral): “We refer the reader to [149] for a more comprehensive survey of recent advances in model compression techniques for DNNs.”
- A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, methodology): “Given the fact that there are many methods to conduct model compression (Deng et al., 2020), e.”
- A Survey on Model Compression for Large Language Models (2023, neutral): “To tackle these issues, a prevalent approach known as model compression [Deng et al., 2020; He et al., 2018] offers a solution.”
- Neural Pruning via Growing Regularization (2020, background): “As deep neural networks advance in recent years LeCun et al. (2015); Schmidhuber (2015), their remarkable effectiveness comes at a cost of rising storage, memory footprint, computing resources and energy consumption Cheng et al. (2017); Deng et al. (2020).”
- Compression of Deep Learning Models for Text: A Survey (2020, background): “While there are other surveys in the broad area of model compression [17], [18], they are either old or focus on computer vision related problems.”
- Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”