Efficient and accurate approximations of nonlinear convolutional networks

By X. Zhang, Jianhua Zou, Xiang Ming, Kaiming He. 2014

270 citations recorded by the source corpus.

This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the “AlexNet” [11], but is 4.7% more accurate.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Spatial Transformer Networks (2015, methodology): “In Table 1 (right), we show the transformations performed for some test cases where a CNN is unable to correctly classify the digit, but a spatial transformer network can.”
The Efficiency Misnomer (2021, methodology): “As an example, tensor factorization is used as one of the common techniques to accelerate the matrix multiplication and used for making neural network models more efﬁcient (Zhang et al., 2015b; Jaderberg et al., 2014).”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “5× model still achieves ∼13× actual speedup under comparable classification accuracy (the theoretical speedup is 18×), which is much faster than previous AlexNet-level models or speedup approaches such as [14, 16, 22, 42, 43, 38].”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, positive): “For example, tensor decomposition is widely used in early works [14,36,37] to accelerate the matrix multiplication.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “In [7, 13, 32, 31, 18, 17], lowrank approximation or tensor decomposition is adopted to speed-up convolutional layers.”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Lowrank factorization methods [6, 7, 16, 27, 45] produce compact low-rank approximations of filter matrices.”
Compression of fully-connected layer in neural network by Kronecker product (2015, background): “Zhang et al. (2014a) considered approximation of multiple layers with nonlinear activations.”
Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, background): “Index Terms—Convolutional Neural Networks, Acceleration, Image Classification, Object Detection F”
Dynamic Network Surgery for Efficient DNNs (2016, methodology): “Following similar ideas, some subsequent methods can provide more significant speedups [5, 22, 14].”
EIE: Efficient Inference Engine on Compressed Deep Neural Network (2016, methodology): “SVD is frequently used to reduce model size [37] [38].”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “In [7, 39, 18], low-rank approximation methods have been utilized to speed up convolutional layers by decomposing the weight matrix into low-rank matrices.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “Methods focus on accelerating pre-trained models include direct pruning [8, 23, 28, 12, 26], low-rank decomposition [6, 19, 44], and quantization [30, 5, 41].”
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “While (Lebedev et al., 2015; Zhang et al., 2015a) reported difﬁculty on ﬁnding a good SGD learning rate, our single learning rate scheduling rule works well for various compressed CNNs.”
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, methodology): “[48] both use PCA to compress the weights of DNN models by up to 5x, albeit with a substantial reduction in the model’s classification accuracy.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “In [36, 37], the authors used generalized singular vector decomposition (GSVD) to decompose an original layer to two approximated layers with reduced computation complexity.”
Design of Efficient Convolutional Layers using Single Intra-channel Convolution, Topological Subdivisioning and Spatial "Bottleneck" Structure (2016, background): “Low rank decomposition has been extensively explored in various fashions [7][8][9][10][11] to obtain moderate efﬁciency improvement.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, methodology): “Most approximate methods of decreasing the CNN computational cost exploit the redundancies of the convolutional kernel using low-rank tensor decompositions [8, 13, 19, 34].”