A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers

By Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, Yanzhi Wang. European Conference on Computer Vision, 2018

471 citations recorded by the source corpus.

Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at this https URL

Subfields: Hardware and Architecture, Media Technology

View Paper: https://www.semanticscholar.org/paper/8f880afbfea328c496d1ab96a123bb57d4b506b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Hardware and Architecture

In Media Technology

Citations in the corpus

AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance (2020, methodology): “To fully execute DNN inference at the edge, many optimizations, such as model architecture search [80, 87, 101], quantization [13,26,43,53,55,92,97], weight compression [15, 37, 58, 61] and graph pruning [93, 96], have been proposed.”
Learn-Prune-Share for Lifelong Learning (2020, methodology): “We use the widely deployed Alternating Direction Method of Multipliers (ADMM) [34], that has been extensively applied in pruning literature [31], [35].”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, methodology): “3k (8%) 330 (66%) 3k (12%) 32k (8%) 950 (19%) ADMM [13] 6.”
A Survey on Model Compression for Large Language Models (2023, unknown): “Pruning can be divided into Unstructured Pruning [Zhang et al. , 2018; Gordon et al. , 2020] and Structured Pruning [Anwar et al. , 2017; Fang et al. , 2023].”
Adversarial Robustness vs. Model Compression, or Both? (2019, methodology): “There are existing irregular pruning work [17, 15, 47, 48] and regular pruning work [20, 42, 49, 25, 29].”
Pruning Filter via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, background): “Many recent works [9, 8, 7, 26, 1, 12, 31] focus on pruning fine-grained weight of filters.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, background): “…pruning (Han et al., 2016b; Srinivas & Babu, 2015; Wen et al., 2016; Tung & Mori, 2018; Aghasi et al., 2017; Carreira-Perpinan & Idelbayev, 2018; Zhang et al., 2018; Dai et al., 2018), where individual neurons are removed, and structured pruning (Li et al., 2017; He et al., 2017; Luo et al.,…”
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning (2020, methodology): “The frameworks and middleware also enable efﬁcient inference at the edge— various NN optimizations, such as quantization [15], [30], [52], [62], [65], [111], [119], weight compression [43], [68], and graph pruning [112], [118] can be employed before the DNNs are deployed.”
Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks (2019, background): “[9, 8, 7, 30, 2, 36, 4] focus on pruning fine-grained weight of filters, which leads to unstructured sparsity in models.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Typical approaches include pruning [12], [13], [14], [15], [16], quantization [17], [18], [19], [20], light-weight neural network design [21], [22], [23], [24], [25], low rank factorization [26], [27] and knowledge distillation [28], [29], [30], [31].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Recently, the ADMM method mentioned in Section III-D has also been used for high-accuracy sparsification [7], [223]–[225].”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “Model compression techniques for DNN models, including element-wise pruning [7, 21, 38], structural pruning [33, 25, 20], quantization [23, 31], and factorization [15, 39, 36, 35], have been extensively studied.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “…model size such as distillation methods (Bucilua et al., 2006; Hinton et al., 2015), reduced bit-precision methods (Hubara et al., 2016; McDonnell, 2018), low-rank decomposition methods (Jaderberg et al., 2014; Denil et al., 2013), and pruning methods (Han et al., 2015a; Zhang et al., 2018b).”
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, background): “Many recent works [11, 10, 9, 36, 1, 15, 41, 4] focus on pruning fine-grained weight of filters.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Liu et al. [128] investigate pruning ResNet20 on CIFAR-10 with ADMM-based [191] one-shot pruning method and ﬁnd that pruning & ﬁne-tuning outperforms LTH (pruning & training from scratch) over various prune ratios.”
Task-Oriented Feature Distillation (2020, methodology): “Various techniques have been proposed to tackle this issue, including pruning [15, 41, 72, 67, 39], quantization [35, 46, 11, 50], lightweight model de-sign [23, 49, 27], and knowledge distillation (KD) [22, 53, 70].”