Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

By Xin Dong, Shangyu Chen, Sinno Jialin Pan. 2017

573 citations recorded by the source corpus.

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.

Subfields: Computer Vision and Pattern Recognition, Neurology, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “[22] Xin Dong, Shangyu Chen, and Sinno Jialin Pan.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “, 2016), redundancy (Mariet & Sra, 2016; Srinivas & Babu, 2015a), per-layer second derivatives (Dong et al., 2017), and energy/computation efficiency (Yang et al.”
Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “…et al., 2018), continuous pruning and re-training (Guo et al., 2016), joint loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer pruning (Dong et al., 2017), fast-switching growth-pruning cycles (Dai et al., 2017), and soft weight-sharing (Ullrich et al., 2017).”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, methodology): “We test our approach for extreme sparsity levels (e.g., up to 99% sparsity on LeNet-5-Caffe) and compare ours against the following various pruning algorithms: LWC (Han et al. (2015)), DNS (Guo et al. (2016)), vi LC (Carreira-Perpiñán & Idelbayev (2018)), SWS (Ullrich et al. (2017)), SVD (Molchano”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 85, 137, 141, 186, 249], and structured pruning [90, 105, 153, 162, 266, 267, 271].”
The Lottery Ticket Hypothesis at Scale (2019, unknown): “(2016); Srinivas & Babu (2015); Dong et al. (2017); Li et al.”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “Several other criterions and settings are proposed for unstructured pruning[6, 39, 31].”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “We emphasize that the techniques we introduce are exact; unlike prior work [6, 39], we do not rely on any approximations.”
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020, background): “structured magnitude weight pruning (also called irregular pruning) (Han et al., 2015; Guo et al., 2016; Dong et al., 2017).”
Data-Independent Structured Pruning of Neural Networks via Coresets (2020, background): “Sparsification, also known as weight pruning, was considered as far back as 1990 [32] but has recently seen more study [9], [14], [15], [17], [47].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “[23] prunes the network locally in a layer by layer manner.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “Singh and Alistarh [37] showed that it can be scaled to DNNs by blockwise approximation, and that it provides better approximations of the loss than K-FAC-based [44], or diagonal [39] approximations [7], leading to more accurate pruning.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “This block-wise simpliﬁcation is is motivated by the observation that Hessians tend to be diagonally-dominant, and has been employed in previous work, e.g. [33].”
DepGraph: Towards Any Structural Pruning (2023, methodology): “be roughly categorized into two schemes, structurual pruning [4, 29, 71] and unstructurual pruning [8, 13, 44].”
Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning (2020, background): “In previous work (Liu et al., 2017; Han et al., 2015; Ding et al., 2019b; Aghasi et al., 2017; Dong et al., 2017; Renda et al., 2020), including the criteria mentioned above, they usually focused on how much the model was compressed, how much performance was restored, the inference efficiency of the”
Data-Independent Neural Pruning via Coresets (2019, background): “, 1990), but has recently seen more study (Lebedev & Lempitsky, 2016; Dong et al., 2017).”
Learning Compact Neural Networks with Regularization (2018, background): “Sparsifying and quantizing neural networks lead to storage efficient compact models that will be building blocks intelligent mobile devices (Han et al., 2015a;b; Courbariaux et al., 2016; Denton et al., 2014; Jin et al., 2016; Dong et al., 2017; Aghasi et al., 2017).”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Subsequent methods [Li et al., 2016, Lin et al., 2017, Dong et al., 2017, Sanh et al., 2020, Lagunas et al., 2021, Zhu and Gupta, 2017] improve on the quality of the pruned models.”