MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

By Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng. IEEE International Conference on Computer Vision, 2019

640 citations recorded by the source corpus.

In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for any pruned structure given the target network. We use a simple stochastic structure sampling method for training the PruningNet. Then, we apply an evolutionary procedure to search for good-performing pruned networks. The search is highly efficient because the weights are directly generated by the trained PruningNet and we do not need any finetuning at search time. With a single PruningNet trained for the target network, we can search for various Pruned Networks under different constraints with little human participation. Compared to the state-of-the-art pruning methods, we have demonstrated superior performances on MobileNet V1/V2 and ResNet. Codes are available on https://github.com/liuzechun/MetaPruning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bd3df472bc848083068a76e9ce2b2ab49543dc78

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “approaches have been proposed to compress and accelerate DNNs, including channel pruning [27, 29], quantization [31, 47, 50], neural architecture search [6, 43], etc.”
Task-Oriented Feature Distillation (2020, methodology): “Various techniques have been proposed to tackle this issue, including pruning [15, 41, 72, 67, 39], quantization [35, 46, 11, 50], lightweight model de-sign [23, 49, 27], and knowledge distillation (KD) [22, 53, 70].”
GhostNet: More Features From Cheap Operations (2019, neutral): “1 MetaPruning-ResNet-50 [36] 1.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, background): “[40] later replaced the reinforcement learning by an evolutionary search algo-rithm.”
Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection (2020, critical): “…on mobile settings such as MobileNetV2 (Sandler et al., 2018) and MobileNetV3 (Howard et al., 2019), and ﬁnd that ﬁnetuning a pruned Mo-bileNetV2/MobileNetV3 gives much better performance than re-training it from a new random initialization, which violates the conclusions of Liu et al. (2019b).”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, methodology): “In structured pruning, parameters are pruned in blocks (pruning row-wise in a weight matrix, or pruning channel-wise in a convolutional filter [5, 84, 89, 98], etc.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Meta-learning is beginning to be applied to this objective as well, such as training gradient generator meta-networks that allow quantized networks to be trained [199], and weight generator meta-networks that allow quantized networks to be trained with gradient [281].”
Lite Transformer with Long-Short Range Attention (2020, methodology): “Recently, AutoML has also been used to automate the model compression and acceleration (He et al., 2018; Yang et al., 2018; Wang et al., 2019; Liu et al., 2019a).”
Dynamic Slimmable Network (2021, positive): “Remarkably, DS-MBNet outperforms the SOTA pruning methods EagleEye [40] and Meta-Pruning [51] by 1.9% and 2.2% .”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, neutral): “Meta pruning is proposed to prune the channels of neural networks by a meta model automatically [59].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “One-shot pruning methods score once and then prune the network to a target prune ratio.”
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution (2020, methodology): “Besides, neural architecture search has also been used in compressing and accelerating neural networks, including pruning [17, 31, 5, 27] and quantization [58, 15, 59, 62].”
Principled Weight Initialization for Hypernetworks (2020, background): “Since then, a big lesson learned by the neural network optimization community is that architecture specific initialization schemes are important to the robust training of deep networks, as shown recently in the case of residual networks (Zhang et al., 2019).”
Provable Filter Pruning for Efficient Neural Networks (2019, methodology): “We envision that besides its immediate use for pruning state-of-the-art models, our approach can also be used as a sub-procedure in other deep learning applications, e.g., for identifying winning lottery tickets (Frankle & Carbin, 2019) and for efﬁcient architecture search (Liu et al., 2019b”
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough (2020, neutral): “9 220M Meta Pruning (Liu et al., 2019) 71.”
On-Device Training Under 256KB Memory (2022, methodology): “Pruning techniques prove to be quite successful for achieving sparsity and reducing model size [30, 31, 49, 32, 51, 50].”
Group Fisher Pruning for Practical Network Compression (2021, neutral): “IE (Molchanov et al., 2019) 71.69 76.18 1.34 - Meta (Liu et al., 2019) 73.40 76.60 1.0 - CURL (Luo & Wu, 2020) 73 using different network structures and FLOPs constraints.”