Accelerating Very Deep Convolutional Networks for Classification and Detection

By X. Zhang, Jianhua Zou, Kaiming He. 2015

853 citations recorded by the source corpus.

This paper aims to accelerate the test-time computation of convolutional neural networks (CNNs), especially very deep CNNs <xref ref-type="bibr" rid="ref1">[1]</xref> that have substantially impacted the computer vision community. Unlike previous methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We develop an effective solution to the resulting nonlinear optimization problem without the need of stochastic gradient descent (SGD). More importantly, while previous methods mainly focus on optimizing one or two layers, our nonlinear method enables an asymmetric reconstruction that reduces the rapidly accumulated error when multiple (e.g., <inline-formula> <tex-math notation="LaTeX">$\ge$</tex-math><alternatives><inline-graphic xlink:type="simple" xlink:href="he-ieq1-2502579.gif"/> </alternatives></inline-formula>10) layers are approximated. For the widely used very deep VGG-16 model <xref ref-type="bibr" rid="ref1">[1]</xref> , our method achieves a whole-model speedup of 4<inline-formula><tex-math notation="LaTeX">$\times$</tex-math> <alternatives><inline-graphic xlink:type="simple" xlink:href="he-ieq2-2502579.gif"/></alternatives></inline-formula> with merely a 0.3 percent increase of top-5 error in ImageNet classification. Our 4<inline-formula><tex-math notation="LaTeX">$\times$ </tex-math><alternatives><inline-graphic xlink:type="simple" xlink:href="he-ieq3-2502579.gif"/></alternatives></inline-formula> accelerated VGG-16 model also shows a graceful accuracy degradation for object detection when plugged into the Fast R-CNN detector <xref ref-type="bibr" rid="ref2">[2]</xref> .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The Efficiency Misnomer (2021, background): “Weight decomposition, regardless of its effect on the quality of the model, can reduce the number of FLOPs in neural networks by 75% (Zhang et al., 2015a).”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “5× model still achieves ∼13× actual speedup under comparable classification accuracy (the theoretical speedup is 18×), which is much faster than previous AlexNet-level models or speedup approaches such as [14, 16, 22, 42, 43, 38].”
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, background): “However, the recent work [7] finds that the decomposition in [36] is even slower on GPU although it reduces FLOPs by 75%.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “In Table 5, we also report the comparison against LANR [31] on VGG-16.”
Network Pruning via Transformable Architecture Search (2019, methodology): “A variety of techniques have been proposed, such as low-rank decomposition [44], weight pruning [11, 26, 10, 9], channel pruning [15, 30], dynamic computation [7, 5] and quantization [20, 1].”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “[53] proposed to factorize layers into 3×3 and 1×1.”
Learning to Prune Filters in Convolutional Neural Networks (2018, methodology): “[34] used Generalized SVD for the non-linearity in networks and achieved promising results in very deep CNNs.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “In contrast, tensor-wise examples include SVD decomposition [9,64], CPdecomposition [28], Tucker decomposition [22], Tensor-Train decomposition [45],”
Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “…Neural Network by Kronecker Product Shuchang Zhou State Key Laboratory of Computer Architecture Institute of Computing Technology Chinese Academy of Sciences, Beijing, China and Megvii Inc. shuchang.zhou@gmail.com Jia-Nan Wu Department of Computer Science & Engineering Shanghai Jiao Tong University”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, methodology): “[53] factorize a layer into 3× 3 and 1× 1 combination, driven by feature map redundancy.”
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “While (Lebedev et al., 2015; Zhang et al., 2015a) reported difﬁculty on ﬁnding a good SGD learning rate, our single learning rate scheduling rule works well for various compressed CNNs.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “In [36, 37], the authors used generalized singular vector decomposition (GSVD) to decompose an original layer to two approximated layers with reduced computation complexity.”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, methodology): “It is also worth noting that some existing work used such strategy in a layer-by-layer greedy way (He et al., 2017; Zhang et al., 2016).”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, methodology): “[28], the third row is the combined method.”
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018, methodology): “In particular, the matrix decomposition of deep CNN tensors is approximated by the product of two low-rank matrices [Jaderberg et al. , 2014; Zhang et al. , 2016; Tai et al. , 2016].”
Extreme Network Compression via Filter Group Approximation (2018, methodology): “As mentioned by [9, 41], deeper layers have less redundancy, which indicates less compression with increasing depth in whole network compression.”