Dynamic Network Surgery for Efficient DNNs

By Yiwen Guo, Anbang Yao, Yurong Chen. 2016

1,158 citations recorded by the source corpus.

Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of $\bm{108}\times$ and $\bm{17.7}\times$ respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Guo et al. (2016) improve on this approach by allowing masked weights to still receive gradient updates, enabling the network to recover from incorrect pruning decisions during optimization.”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, neutral): “Popular non-magnitude based techniques include [26, 17, 34].”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Guo et al. (2016) restore pruned connections as they become relevant again.”
Weight Agnostic Neural Networks (2019, neutral): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Dynamic Model Pruning with Feedback (2020, methodology): “Their results (i.e., moderate sparsity level with minor quality loss) made them the standard method for network pruning and led to several variations (Alvarez & Salzmann, 2016; Guo et al., 2016; Alvarez & Salzmann, 2017; Carreira-Perpinán & Idelbayev, 2018).”
Recent advances in convolutional neural networks (2015, methodology): “[182] extend the magnitude-based approach to allow restoration of the pruned weights in the previous iterations, with tightly coupled pruning and retraining stages, for greater model compression.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “In recent years, impressive results have been achieved using the magnitude of weight as the criterion (Han et al., 2016) as well as other variations (Guo et al., 2016).”
You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, methodology): “The early works of pruning focus on removing unimportant connections (LeCun et al. (1990); Hassibi & Stork (1993); Han et al. (2015); Guo et al. (2016)).”
MaskConnect: Connectivity Learning by Gradient Descent (2018, positive): “Several authors have proposed learning connectivity by pruning unimportant weights from the network [26,27,28,29,30].”
Learning to Prune Filters in Convolutional Neural Networks (2018, background): “[9] proposed a dynamic network surgery algorithm to compress CNNs by making them more sparse.”
Connectivity Learning in Multi-Branch Networks (2017, methodology): “Our approach is also related to methods that learn connectivity by pruning unimportant weights from the network (LeCun et al., 1989; Han et al., 2015a;b; Guo et al., 2016; Han et al., 2016).”
Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “Most recent works on sparse neural networks use pruning (Han et al., 2015b), elastic net regularization (Lebedev & Lempitsky, 2015; Liu et al., 2015; Scardapane et al., 2016; Wen et al., 2016) or composite techniques (Han et al., 2015a; Guo et al., 2016; Ullrich et al., 2017).”
Bayesian Compression for Deep Learning (2017, methodology): ““DC” corresponds to Deep Compression method introduced at [25], “DNS” to the method of [21] and “SWS” to the Soft-Weight Sharing of [70].”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Sparse connection eliminates connections between neurons [17, 33, 29, 15, 14].”
Soft Weight-Sharing for Neural Network Compression (2017, background): “We compare our compression scheme with Han et al. (2015a) and Guo et al. (2016) in Table 1.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “[7] proposed dynamic network surgery to prune parameters during training.”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “, 2017), Dynamic Network Surgery (Guo et al., 2016) and Sensitivity Driven Regularization (Tartaglione et al.”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, methodology): “[10] extend the magnitude-based methods to allow the restoration of the pruned weights in the previous iterations, with tightly coupled pruning and global fine-tuning stages, for greater model compression.”