Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
By Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin. 2015
949 citations recorded by the source corpus.
Although the latest high-end smartphone has powerful CPU and GPU, running deeper convolutional neural networks (CNNs) for complex tasks such as ImageNet classification on mobile devices is challenging. To deploy deep CNNs on mobile devices, we present a simple and effective scheme to compress the entire CNN, which we call one-shot whole network compression. The proposed scheme consists of three steps: (1) rank selection with variational Bayesian matrix factorization, (2) Tucker decomposition on kernel tensor, and (3) fine-tuning to recover accumulated loss of accuracy, and each step can be easily implemented using publicly available tools. We demonstrate the effectiveness of the proposed scheme by testing the performance of various compressed CNNs (AlexNet, VGGS, GoogLeNet, and VGG-16) on the smartphone. Significant reductions in model size, runtime, and energy consumption are obtained, at the cost of small loss in accuracy. In addition, we address the important implementation level issue on 1?1 convolution, which is a key operation of inception module of GoogLeNet as well as CNNs compressed by our proposed scheme.
Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mathematics
- Qiang Yang
- Timothy M. Hospedales
- Jean Kossaifi
- Yongxin Yang
- Yi-Zhe Song
- Da Li
- Zachary Chase Lipton
- Yang He
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Alan L. Yuille
- Jitendra Malik
In Efficiency & Compression
Citations in the corpus
- Aggregated Residual Transformations for Deep Neural Networks (2016, methodology): “Decomposition (at spatial [6, 18] and/or channel [6, 21, 16] level) is a widely adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them.”
- Data-Efficient Structured Pruning via Submodular Optimization (2022, neutral): “, 2015), while another set of methods use low-rank approximation or tensor factorization (Denil et al., 2013; Sainath et al., 2013; Jaderberg et al., 2014; Denton et al., 2014; Lebedev et al., 2015; Kim et al., 2015; Zhang et al., 2016; Su et al., 2018).”
- What is the State of Neural Network Pruning? (2020, methodology): “…entirely fair to methods designed to optimize other metrics, such as power consumption (Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015), memory bandwidth usage (Peng et al., 2018; Kim et al., 2015), or fine-tuning time (Dubey et al., 2018; Yamamoto & Maeno, 2018;…”
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Tensor factorization [30, 15, 27, 35] decomposes weights into light-weight pieces, for example [51, 11, 14] proposed to accelerate the fully connected layers with truncated SVD; Jaderberg et al .”
- Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “In contrast, tensor-wise examples include SVD decomposition [9,64], CP-decomposition [28], Tucker decomposition [22], Tensor-Train decomposition [45], Tensor-Ring decomposition [68], T-Basis [44], etc.”
- Compressing Convolutional Neural Networks in the Frequency Domain (2015, background): “[28] investigate general tensor decomposition with rank selection for compressing the entire network.”
- Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Tensor factorization [22, 28, 13, 24] decompose weights into several pieces.”
- NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “Layer-decomposition-based algorithms [13, 23] exploit matrix decomposition to reduce the number of operations.”
- Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning (2016, neutral): “Our approach is in line with existing work in structured pruning [1], while largely orthogonal to fine-grained pruning of fully-connected layers [3] and speedup by reduced precision [2] or tensor decomposition [6].”
- NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “Finally, we benchmark the pruning results and compare to existing methods such as [11, 18, 33, 22].”
- Deeper, Broader and Artier Domain Generalization (2017, positive): “In our case we exploit this idea both for compression – but across rather than within domains [13], as well as for crossdomain (rather than cross-category [31]) knowledge sharing.”
- Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, methodology): “The proposed energy-efficient pruning algorithm can be combined with other techniques to further reduce the energy consumption, such as bitwidth reduction of weights or feature maps [15, 25, 26], weight sharing and Huffman coding [8], student-teacher learning [27], filter decomposition [28, 29] and ”
- Demystifying Parallel and Distributed Deep Learning (2018, background): “,244,254,255] Hierarchical PS[88,107,253] Decentralized[9,54,121,155] Compression Quantization[5,35,50,51,55,63,87,90,109,132,151,202,210,237,265] Sparsification[3,32,67,158,206,214,224] Other Methods[41,106,111,129,150,243,254] Training Distribution Model Consolidation Ensemble Learning[114,148,217”
- BranchyNet: Fast inference via early exiting from deep neural networks (2016, methodology): “Kim et al. (2015) use a Tucker decomposition (a tensor extension of SVD) to extract shared information between convolutional layers and perform rank selection [11].”
- Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups (2016, methodology): “More general methods have used reduced precision number representations [16] or compression of previously trained models [17], [18].”
- Joint architecture and knowledge distillation in CNN for Chinese text recognition (2019, methodology): “In [27,28] , Tucker decomposition is used to achieve compression.”
- Interleaved Group Convolutions for Deep Neural Networks (2017, background): “Tensor decomposition usually finds a low-rank tensor to approximate the tensor through decomposition along the spatial di-mension [6, 17], or the input and output channel dimensions [6, 19, 17].”
- Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, methodology): “larization. Other approaches include combining parameters with correlated weights (Srinivas & Babu, 2015), reducing precision (Gupta et al., 2015; Rastegari et al., 2016) or tensor decomposition (Kim et al., 2015). These approaches usually require a separate training procedure or significant fine-”
- Tensor Regression Networks (2017, methodology): “Kim et al. (2015) take a pre-trained network and apply tensor (Tucker) decomposition on the convolutional kernel tensors and then fine-tune the resulting network.”