PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions

By Mikhail Figurnov, Aizhan Ibraimova, Dmitry P. Vetrov, Pushmeet Kohli. 2015

140 citations recorded by the source corpus.

We propose a novel approach to reduce the computational cost of evaluation of convolutional neural networks, a factor that has hindered their deployment in low-power devices such as mobile phones. Inspired by the loop perforation technique from source code optimization, we speed up the bottleneck convolutional layers by skipping their evaluation in some of the spatial positions. We propose and analyze several strategies of choosing these positions. We demonstrate that perforation can accelerate modern convolutional networks such as AlexNet and VGG-16 by a factor of 2x - 4x. Additionally, we show that perforation is complementary to the recently proposed acceleration method of Zhang et al. [28].

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

What is the State of Neural Network Pruning? (2020, background): “As others have noted (Lebedev et al., 2014; Figurnov et al., 2016; Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015; Wen et al., 2016; Luo et al., 2017; He et al., 2018b), these metrics are far from perfect.”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
Fast ConvNets Using Group-Wise Brain Damage (2015, background): “rly to [26], the work [11] uses sparsiﬁ- cation to reduce the number of parameters in the memory-bound scenario. Their goal is thus to save memory rather than to attain acceleration. In the report of [14], the output of the convolution is computed at a sparsiﬁed set of locations with the gaps being ”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “Finally, we benchmark the pruning results and compare to existing methods such as [11, 18, 33, 22].”
Adaptive Neural Networks for Efficient Inference (2017, background): “More recently, researchers have designed spatially adaptive networks (Figurnov et al., 2016a; Bengio et al., 2015) where nodes in a layer are selectively activated.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “s, which can be exploited to increase the speed of the evaluation process. In [22], the authors used the groupsparsity regularizer to prune the convolutional kernel tensor in a group-wise fashion. In [7], they increased the speed of convolutional layers by skipping their evaluation at some ﬁxed spat”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “This transformation can be implemented efficiently using the perforated convolutional layer [10].”
Structured Bayesian Pruning via Log-Normal Multiplicative Noise (2017, methodology): “Spatial Skipped Convolutions [3] and Spatially Adaptive Computation Time [2] that propose different ways to reduce the number of computed convolutions, Binary Networks [18] that achieve speedup by using only 1 bit to store a single weight of a DNN, Low-Rank Expansions [6] that use low-rank filter ap”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, methodology): “In this context, existing approaches can be roughly grouped into two categories: (i) parameter pruning and sharing [7, 9, 14, 16, 31, 32, 38, 55], which aims to remove the least informative parameters, yielding an arbitrary compact network with information loss; and (ii) low-rank matrix factorizatio”
Insights on representational similarity in neural networks with canonical correlation (2018, background): “ains largely robust. SVCCA performs better than the unweighted mean but less well than the projection weighting. not rely on the full dimensionality of a layer to represent high performance solutions [12, 6, 1, 20, 15, 21, 14]. As a result, the mean correlation coefﬁcient will typically underestimat”
Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, methodology): “Independently of our work, Figurnov et al. (2016) came up with similar metric based on the Taylor expansion, called impact, to evaluate importance of spatial cells in a convolutional layer.”
Zero-Cost Proxies for Lightweight NAS (2021, background): “They build on the work of Molchanov et al. (2017) and Figurnov et al. (2016).”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, unknown): “PerforatedCNN [10] generates masks by several deterministic sampling methods.”
Nonparametric Neural Networks (2017, background): “Alvarez & Salzmann (2016) utilize an (cid:96) 2 penalty to eliminate units and Molchanov et al. (2017) compare a variety of strategies, whereas Figurnov et al. (2016) focuses on thinning convolutional layers in the spatial dimensions.”
Single-Shot Freestyle Dance Reenactment (2020, methodology): “Finally, the decoder u employs fractional strided convolutions [12], ReLU activations, and instance normalization [40].”
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries (2019, background): “Other complementary approaches include skipping some layers at some pixels [17] and early stopping of network computation for some spatial regions of the image [16, 33].”
Towards Evolutionary Compression (2018, background): “Therefore, more flexible approaches [10, 18, 46] have been developed to directly discard redundant filters in a given convolutional layer:”