EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

By Chaoqi Wang, Roger Baker Grosse, Sanja Fidler, Guodong Zhang. 2019

161 citations recorded by the source corpus.

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structured pruning methods in this basis. As opposed to existing Hessian-based pruning algorithms which do pruning in parameter coordinates, our method works in the KFE where different weights are approximately independent, enabling accurate pruning and fast computation. We demonstrate empirically the effectiveness of the proposed method through extensive experiments. In particular, we highlight that the improvements are especially significant for more challenging datasets and networks. With negligible loss of accuracy, an iterative-pruning version gives a 10$\times$ reduction in model size and a 8$\times$ reduction in FLOPs on wide ResNet32.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

WoodFisher: Efficient second-order approximations for model compression (2020, unknown): “, 2019), pruning (Zeng and Urtasun, 2019; Wang et al., 2019), reinforcementlearning (Wu et al.”
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “In this section, we present the results of the ablation study for Level 2 methods from Zhang et al. (2016), and compare it to channel pruning suggested by He et al.”
How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “Second-order pruning methods, e.g. [16, 24, 42, 63, 65] augment this basic metric with second-order information, which can lead to higher accuracy of the resulting pruned models, relative to GMP. Regularization Methods are usually applied during model training, via sparsity-promoting mechanisms.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “We ﬁrst show that our framework yields signiﬁcantly better solutions for the layer-wise compression problem, which leads to higher-accuracy end-to-end compressed models for both pruning and quantization, relative to the corresponding state-of-the-art techniques, often by signiﬁcant margins.”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “As a result, K-FAC has found several applications in the last few years in: optimization [38, 39], pruning [21, 22], reinforcement-learning [51], etc.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “We can get the following equation based on [98], where ∂ L ∂w is the ﬁrst-order gradient of loss function w.r.t. weights, and H is the Hessian matrix containing second-order derivatives.”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Weights quantization [25, 17, 31, 45] and decomposition [5, 23, 43, 9] aim to accelerate and reduce the memory footprint of CNNs by replacing large matrices operations with their lightweight approximations.”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, background): “There are several ways to remove the structure, such as l1-dependent pruning [15, 64], first-order importance estimation [17], hessian-based estimation [20, 51] or the optimal brain surgeon [23, 20].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “…of the loss function is early used in [161], [164] for removing unimportant weights and gradually exploited in many subsequent methods ([62], [67], [163], [165], [166]), which includes the ﬁrst-order (gradient) term, the second-order (Hessian) term, and the higher-order terms are neglected.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “Singh and Alistarh [37] showed that it can be scaled to DNNs by blockwise approximation, and that it provides better approximations of the loss than K-FAC-based [44], or diagonal [39] approximations [7], leading to more accurate pruning.”