Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges

By Yu Cheng, Duo Wang, Pan Zhou, Zhang Tao. 2018

686 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Beyond human-level accuracy: computational challenges in deep learning (2019, background): “ts, but some algorithmic optimizations may be promising to chip away at this gap. Model compression or distillation, and low-precision or sparse computation may reduce model or activation tensor size [9], and reduce memory requirements by 1:5–10. Currently, many challenges exist to use these techniq”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “Note that these operators could easily be extended by sophisticated methods for compressing neural networks (Han et al., 2016; Cheng et al., 2018).”
Stronger generalization bounds for deep nets via a compression approach (2018, background): “The idea of compressing trained deep nets is very popular for low-power applications; for a survey see [Cheng et al., 2018].”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “Therefore, model compression approaches [27], [28] have attracted much attention for deploying DNN models at network edges.”
Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation (2020, background): “CNN compression [24] aims to improve the efficiency of CNNs by compressing its parameters.”
Distilling Neural Networks for Greener and Faster Dependency Parsing (2020, background): “For a more thorough survey of current neural network compression methods see Cheng et al. (2018).”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Four prominent compression techniques, such as parameter pruning, low-rank approximation, compact CNN design, and knowledge distillation (KD), are summarized in [6].”
Distilling Knowledge From a Deep Pose Regressor Network (2019, background): “-1 or 1), binary networks can dramatically reduce both computation time and memory consumption at the cost of significant reduction in accuracy [5].”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, positive): “increasingly complex, model compression techniques are becoming more important to reduce service latency and resource consumption in provisioning AIGC services [230].”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, methodology): “For other techniques, we recommend the comprehensive review in [191].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “…without compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed descriptions and comparisons for the individual or…”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, background): “1) Edge Device Distributed Inference: Enormous efforts on TinyML with DL model compression and neural network architecture search have been conducted to enable low-latency and energy-efﬁcient model inference on a single device with limited storage and computation resources [220].”