DepGraph: Towards Any Structural Pruning

By Gongfan Fang, Xinyin Ma, Mingli Song, Michael B. Mi, Xinchao Wang. Computer Vision and Pattern Recognition, 2023

518 citations recorded by the source corpus.

Structural pruning enables model acceleration by removing structurally-grouped parameters from neural networks. However, the parameter-grouping patterns vary widely across different models, making architecture-specific pruners, which rely on manually-designed grouping schemes, non-generalizable to new architectures. In this work, we study a highly-challenging yet barely-explored task, any structural pruning, to tackle general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. The most prominent obstacle towards this goal lies in the structural coupling, which not only forces different layers to be pruned simultaneously, but also expects all removed parameters to be consistently unimportant, thereby avoiding structural issues and significant performance degradation after pruning. To address this problem, we propose a general and fully automatic method, Dependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a simple norm-based criterion, the proposed method consistently yields gratifying performances.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/da075ad0ec2c88335af85602a76a33e034536896

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. J. Kolter
2. Yu Li
3. Mingjie Sun
4. Anna Bair
5. Zhuang Liu
6. Chun Yuan
7. Zhendong Yang
8. Ailing Zeng

Citations in the corpus

• A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Structured pruning methods (Liu et al., 2017; Molchanov et al., 2019; Fan et al., 2020; Shen et al., 2022; Xia et al., 2022; Fang et al., 2023; Nova et al., 2023), sometimes referred to as activation pruning (Gale et al.”
• Effective Whole-body Pose Estimation with Two-stages Distillation (2023, methodology): “The basic compression tools comprise distillation [14], pruning [8], and quantiar X iv :2 30 7.”
• A Survey on Model Compression for Large Language Models (2023, unknown): “Pruning can be divided into Unstructured Pruning [Zhang et al. , 2018; Gordon et al. , 2020] and Structured Pruning [Anwar et al. , 2017; Fang et al. , 2023].”