Learning Structured Sparsity in Deep Neural Networks
By Wujie Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Helen Li. 2016
2,511 citations recorded by the source corpus.
High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in this https URL
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Alan L. Yuille
- Jitendra Malik
In Efficiency & Compression
In Optimization & Training
- Sebastian U. Stich
- Peter Richtárik
- Quoc V. Le
- Martin Jaggi
- Ji Liu
- Yoshua Bengio
- Mingxing Tan
- Jason D. Lee
Citations in the corpus
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, background): “Structure pruning [37, 18, 17, 27, 12, 24, 23, 7, 13, 10] and growing [9, 38, 27, 7] were proposed for different goals, such as efficient inference [37, 18, 17, 12, 24, 23, 7, 13, 10], lifelong learning [38] and model adaptation [9, 27].”
- Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “( ii ) Unlike layerwise compression models (Denil et al., 2013; Wen et al., 2016), we operate in the entire parameter space, which could work better or worse, depending on the network.”
- Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “However, it has been more widely adopted in computer vision and applied to convolutional network to remove filters (Li et al., 2016; Wen et al., 2016), channels (He et al.”
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, methodology): “Pruning network connections [6, 7] or channels [38] reduces redundant connections in a pre-trained model while maintaining performance.”
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018, neutral): “Such discrepancy has been noticed in previous works [7,19,24,30].”
- What is the State of Neural Network Pruning? (2020, background): “As others have noted (Lebedev et al., 2014; Figurnov et al., 2016; Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015; Wen et al., 2016; Luo et al., 2017; He et al., 2018b), these metrics are far from perfect.”
- Similarity-Preserving Knowledge Distillation (2019, methodology): “Weight pruning methods [11, 20, 22, 24, 33, 35, 39] remove unimportant weights from the network, sparsifying the network connectivity structure.”
- Recent advances in convolutional neural networks (2015, methodology): “ve. HashNets may be used together with pruning to give even better parameter savings. 4.5. Sparse Convolution Recently, several attempts have been made to sparsify the weights of convolutional layers [190, 191]. Liu et al . [190] consider sparse representations of the basis lters, and achieve 90% sp”
- Model compression via distillation and quantization (2018, background): “is end, various elegant compression techniques have been 1 arXiv:1802.05668v1 [cs.NE] 15 Feb 2018 Published as a conference paper at ICLR 2018 proposed, e.g. Han et al. (2015); Iandola et al. (2016); Wen et al. (2016); Gysel et al. (2016); Mishra et al. (2017), which combine quantization, weight sha”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “Structured pruning, e.g. channel pruning (Wen et al., 2016; He et al., 2017), was widely adopted as a hardware-friendly compression means (Gui et al., 2016).”
- Network Pruning via Transformable Architecture Search (2019, methodology): “Some methods implicitly find a data-specific architecture [39, 2, 12], by automatically determining the pruning and compression ratio of each layer.”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “However, it can be different for W or constructed to represent the structural sparsity for Convolutional kernels in NN aiming for Network Compression, which is related to Bayesian compression (Louizos et al., 2017) and structural sparsity compression (Wen et al., 2016).”
- Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “Some other methods use group LASSO [48] to encode specific structures during regularisation, such as channel or filter level pruning [30, 41, 18].”
- You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, methodology): “Based on this work, several methods have been proposed to accelerate the search process by limiting the search space (Zoph et al. (2018)), early stopping with performance prediction (Zhong et al.”
- Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, background): “…et al., 2018; Luo et al., 2017; He et al., 2017; Molchanov et al., 2017; He et al., 2018b; Yang et al., 2018a; Jiang et al., 2018; He et al., 2018a; Hu et al., 2016; Liu et al., 2017; Li et al., 2017; Lin et al., 2018; Wen et al., 2016) given its fine-grained control capability over the constraints.”
- MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “Except for the dropout, all other hyperparameters used at training were identical for all DNNs.”
- Lifelong Learning with Dynamically Expandable Networks (2017, positive): “group defined on the incoming weights for each neuron. For convolutional layers, we defined each group as the activation map for each convolutional filter. This group sparsity regularization was used in Wen et al. (2016) and Alvarez & Salzmann (2016) to find the right number of neurons for a full ne”
- Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…network to convergence, scores the weight parameters (usually based on l 1 /l 2 -norms), prunes out low-scored parameters, and fine-tunes the pruned networks (LeCun et al., 1990; Han et al., 2015; Li et al., 2017; Lebedev & Lempitsky, 2016; Wen et al., 2016; He et al., 2017; Frankle & Carbin, 2019).”
- Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, methodology): “The group Lasso [40] regularization, which was favorably used in model compression [4, 37, 38, 29], has been also adopted for continual learning in [39].”