Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

By Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, David S. Doermann. Computer Vision and Pattern Recognition, 2019

578 citations recorded by the source corpus.

Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scale factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and ImageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88% Top-5 error and results in a factor of 3.7x speedup. This significantly outperforms state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dee5d062e70250572e50cda25f08d6a2c02b2bab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Yi Yang
4. Ming-Hsuan Yang
5. Ling Shao
6. Xiaogang Wang
7. Alan L. Yuille
8. Jiashi Feng

Citations in the corpus

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have been widely used in many other areas such as malware detection [434], chess game playing [435], steganography [436]–[439], privacy-preserving [440]–[442], social robot [443], and network pruning [444], [445].”
• Highlight Every Step: Knowledge Distillation via Collaborative Teaching (2019, methodology): “Therefore, many methods [7]–[11] are proposed to reduce the deep neural network’s computational”
• A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “-CGAN) [175] for video question answering. 3.4 Miscellaneous GAN applications GANs have also been used in many other domains such as malware detection [176], chess game playing [177], network pruning [178, 179], spatial representation learning [180], mobile user profiling [181], data augmentation [1”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Generative Adversarial Learning (GAL) [81] jointly prunes structures and adopts a Generative Adversarial Network (GAN) to achieve label-free learning.”
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation (2020, background): “Alternatively, sparsity learning techniques identify what to prune in conjunction with network training, through constraints that zero out some filters [36], cause some filters to become redundant and removable [6], scale some filter or block outputs to zero [23], or sparsify batch normalization sca”
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020, methodology): “Pruning [11, 23, 39, 25, 45, 47, 24] is used to reduce the redundant parameters in neural networks that are insensitive to the model performance.”
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, methodology): “Some filter pruning approaches [23, 25, 16, 27, 7, 33, 39, 37, 46, 14, 18, 22] need to utilize training data to determine the pruned filters.”
• Data-Independent Structured Pruning of Neural Networks via Coresets (2020, methodology): “An attempt of global pruning was done in [38]; it employs the GAN framework to make the output of the pruned and the baseline networks follow the same distribution.”
• Training convolutional neural networks with cheap convolutions and online distillation (2019, methodology): “, network architecture search [46, 47, 48, 49, 50, 51], parameter quantization [14, 15, 13, 52, 53], network pruning [9, 10, 54, 11, 12, 55, 56, 57, 58] and low-rank decomposition [59, 4, 7, 6, 5, 8].”
• Discrimination-Aware Network Pruning for Deep Model Compression (2020, methodology): “On ILSVRC-12, we compare DCP with SFP [21], GAL [41], Taylor-FO-BN [51], SPP [76], CP [24], GDP [40], FPGM [23], ThiNet [50], SSR-L2 [39], NetAdapt [79], AMC [22], and MetaPruning [46].”
• HRank: Filter Pruning Using High-Rank Feature Map (2020, methodology): “, the mask-based scheme, the modified loss even requires specialized optimizers [16, 23], which affects the flexibility and ease of using approaches based on adaptive importance.”