Fast ConvNets Using Group-Wise Brain Damage

By Vadim Lebedev, Victor S. Lempitsky. 2015

461 citations recorded by the source corpus.

We revisit the idea of brain damage, i.e. the pruning of the coefficients of a neural network, and suggest how brain damage can be modified and used to speedup convolutional layers in ConvNets. The approach uses the fact that many efficient implementations reduce generalized convolutions to matrix multiplications. The suggested brain damage process prunes the convolutional kernel tensor in a group-wise fashion. After such pruning, convolutions can be reduced to multiplications of thinned dense matrices, which leads to speedup. We investigate different ways to add group-wise prunning to the learning process, and show that severalfold speedups of convolutional layers can be attained using group-sparsity regularizers. Our approach can adjust the shapes of the receptive fields in the convolutional layers, and even prune excessive feature maps from ConvNets, all in data-driven way.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, background): “Structure pruning [37, 18, 17, 27, 12, 24, 23, 7, 13, 10] and growing [9, 38, 27, 7] were proposed for different goals, such as efficient inference [37, 18, 17, 12, 24, 23, 7, 13, 10], lifelong learning [38] and model adaptation [9, 27].”
Pruning Filters for Efficient ConvNets (2016, background): “Concurrently with our work, there is a growing interest in training compact CNNs with sparse constraints (Lebedev & Lempitsky (2016); Zhou et al. (2016); Wen et al. (2016)).”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “In Table 4, we report the performance under several ( C ′ s , K ) settings, comparing with two baseline methods, CPD [17] and GBD [18].”
What is the State of Neural Network Pruning? (2020, background): “In other cases, it means the fraction of parameters or FLOPs removed (Han et al., 2015; Lebedev & Lempitsky, 2016; Yao et al., 2018).”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “Within the context of reducing model size by removing redundant connections, several recent works (Anwar et al., 2015; Lebedev and Lempitsky, 2015; Li et al., 2016; Changpinyo et al., 2017) propose techniques to prune and induce sparsity in a structured way, motivated primarily by the desire to…”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, background): “First, network pruning is a way to reduce network complexity by pruning the redundant and non-informative weights in a pretrained model [26, 17, 7].”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “Some other methods use group LASSO [48] to encode specific structures during regularisation, such as channel or filter level pruning [30, 41, 18].”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “Except for the dropout, all other hyperparameters used at training were identical for all DNNs.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…network to convergence, scores the weight parameters (usually based on l 1 /l 2 -norms), prunes out low-scored parameters, and ﬁne-tunes the pruned networks (LeCun et al., 1990; Han et al., 2015; Li et al., 2017; Lebedev & Lempitsky, 2016; Wen et al., 2016; He et al., 2017; Frankle & Carbin, 2019).”
Variational Dropout Sparsifies Deep Neural Networks (2017, background): “Most recent works on sparse neural networks use pruning (Han et al., 2015b), elastic net regularization (Lebedev & Lempitsky, 2015; Liu et al., 2015; Scardapane et al., 2016; Wen et al., 2016) or composite techniques (Han et al., 2015a; Guo et al., 2016; Ullrich et al., 2017).”
Learning Structured Sparsity in Deep Neural Networks (2016, background): “A most related parallel work is Group-wise Brain Damage [17], which is a subset (i.”
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (2017, background): “Lebedev and Lempitsky [19] explored group-sparse convolution by introducing the group-sparsity regularization to the loss function, then some entire groups of weights would shrink to zeros, thus can be removed.”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Sparse connection eliminates connections between neurons [17, 33, 29, 15, 14].”
Rethinking the Value of Network Pruning (2018, methodology): “Group sparsity is also widely used to smooth the pruning process after training (Wen et al., 2016; Alvarez & Salz-mann, 2016; Lebedev & Lempitsky, 2016; Zhou et al., 2016).”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Network parameter reduction: Considerable efforts have been devoted to reducing the number of parameters in an existing network [13, 12, 17, 36, 7, 8, 24, 27, 38, 51, 49, 4, 35, 28, 45, 10, 46].”
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science (2017, neutral): “Currently, all state-of-the-art deep learning implementations are based on very well-optimized dense matrix multiplications on graphics processing units (GPUs), while sparse matrix multi-plications are extremely limited in performance 50,51 .”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “Generally, most of these sparsity-based methods [22, 25, 31] integrate the sparsity property as a regularizer into the learning of parameters, which usually harms the performance of network.”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, background): “Some recent efforts proposed to impose structured sparsity on those vector components motivated from the implementation perspective on specialized hardware (Wen et al., 2016; Zhou et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016).”
Importance Estimation for Neural Network Pruning (2019, methodology): “We compare against pruning methods that use various heuristics, such as weight magnitude [21, 23], magnitude of the batch-norm scale, BN scale [6, 24, 32], and output-based heuristics (Taylor expansion applied to layer outputs) [27].”