Data-Driven Sparse Structure Selection for Deep Neural Networks

By Zehao Huang, Naiyan Wang. 2017

617 citations recorded by the source corpus.

Deep convolutional neural networks have liberated its extraordinary power on various tasks. However, it is still very challenging to deploy state-of-the-art models into real-world applications due to their high computational complexity. How can we design a compact and effective network without massive experiments and expert knowledge? In this paper, we propose a simple and effective framework to learn and prune deep models in an end-to-end manner. In our framework, a new type of parameter -- scaling factor is first introduced to scale the outputs of specific structures, such as neurons, groups or residual blocks. Then we add sparsity regularizations on these factors, and solve this optimization problem by a modified stochastic Accelerated Proximal Gradient (APG) method. By forcing some of the factors to zero, we can safely remove the corresponding structures, thus prune the unimportant parts of a CNN. Comparing with other structure selection methods that may need thousands of trials or iterative fine-tuning, our method is trained fully end-to-end in one training pass without bells and whistles. We evaluate our method, Sparse Structure Selection with several state-of-the-art CNNs, and demonstrate very promising results with adaptive depth and width selection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “However, it has been more widely adopted in computer vision and applied to convolutional network to remove filters (Li et al., 2016; Wen et al., 2016), channels (He et al., 2017), or residual blocks (Huang et al., 2018; Huang & Wang, 2018).”
What is the State of Neural Network Pruning? (2020, background): “…(Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015), memory bandwidth usage (Peng et al., 2018; Kim et al., 2015), or ﬁne-tuning time (Dubey et al., 2018; Yamamoto & Maeno, 2018; Huang & Wang, 2018; He et al., 2018a), and we consider this a limitation of our analysis.”
You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, methodology): “Fortunately, a recently proposed method Sparse Structure Selection (SSS) (Huang & Wang (2018)) solved this challenging problem by modifying a theoretically sound optimization method Accelerated Proximal Gradient (APG) method, by reformulating it to avoid redundant forward and backward in calculating”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, neutral): “This technique was also discovered by a recent work [20] and similar ideas appear elsewhere [14].”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, neutral): “5 SSS [25] X 2.”
Rethinking the Value of Network Pruning (2018, methodology): “Sparse Structure Selection (Huang & Wang, 2018) also uses sparsiﬁed scaling factors to prune structures, and can be seen as a generalization of Network Slimming.”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, methodology): “A recent work by Huang & Wang (2017) used a similar technique as ours to remove unimportant residual modules in ResNet by introducing extra scaling factors to the original network.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “Performance of two structured pruning methods, ThiNet and Sparse Structure Selection (SSS), are also listed for comparison (below the double line, see Appendix C for discussion of their relevance); note the difference in parameter counts. used when training using our method.”
Importance Estimation for Neural Network Pruning (2019, methodology): “ResNet-34 ResNet-50 ResNet-101 No pruning Taylor-FO-BN (Ours) SSS [17] Li et al.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 85, 137, 141, 186, 249], and structured pruning [90, 105, 153, 162, 266, 267, 271].”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “More recently, Huang and Wang [23] propose a method for selecting a subset of residual blocks to be executed based on a sparsity constraint.”
Channel Selection Using Gumbel Softmax (2020, methodology): “Our work diﬀers from [21] by using Gumbel Softmax to integrate the sparsity constraint into an additive loss which can be trained by any optimization technique; we use unmodiﬁed stochastic gradient descent with momentum (SGD), the standard technique for training classiﬁcation.”
Search for Better Students to Learn Distilled Knowledge (2020, methodology): “To overcome this challenge, [15] reformulates the original APG to avoid redundant forward and backward pass in calculating the gradients.”
Deep Mixture of Experts via Shallow Embedding (2018, methodology): “Similarly, DeepMoE outperforms the state-of-the-art static channel pruning results [19, 11, 17, 13] on both ImageNet shown in Tab.”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, methodology): “Similarly, STR can be extended and applied to methods for channel pruning and block sparsity (Liu et al., 2019; He et al., 2017; Huang & Wang, 2018) in DNNs.”
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, methodology): “In terms of the granularity of sparsity, current network pruning methods can be divided into unified (Li et al. (2016); He et al. (2017); Luo et al. (2017)) or dynamic strategy (Suau et al. (2018); Liu et al. (2018a); Molchanov et al. (2016); Huang & Wang (2018)).”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, neutral): “51 × SSS (Huang & Wang, 2018) 74.18% 91.91% 1 .”