Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks

By Ting-Wu Chin, Cha Zhang, Diana Marculescu. 2018

49 citations recorded by the source corpus.

Resource-efficient convolution neural networks enable not only the intelligence on edge devices but also opportunities in system-level optimization such as scheduling. In this work, we aim to improve the performance of resource-constrained filter pruning by merging two sub-problems commonly considered, i.e., (i) how many filters to prune for each layer and (ii) which filters to prune given a per-layer pruning budget, into a global filter ranking problem. Our framework entails a novel algorithm, dubbed layer-compensated pruning, where meta-learning is involved to determine better solutions. We show empirically that the proposed algorithm is superior to prior art in both effectiveness and efficiency. Specifically, we reduce the accuracy gap between the pruned and original networks from 0.9% to 0.7% with 8x reduction in time needed for meta-learning, i.e., from 1 hour down to 7 minutes. To this end, we demonstrate the effectiveness of our algorithm using ResNet and MobileNetV2 networks under CIFAR-10, ImageNet, and Bird-200 datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Yang
2. Yang He
3. Linchao Zhu
4. Ping Liu
5. Yuan Xie
6. Guoqi Li
7. Luping P. Shi
8. Song Han

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Linchao Zhu
3. Song Han
4. Yang He
5. Yuan Xie
6. Luc Van Gool
7. Ping Liu
8. Dacheng Tao

In Efficiency & Compression

1. Yang He
2. Yi Yang
3. Ping Liu
4. Yuan Xie
5. Guoqi Li
6. Luping P. Shi
7. By Lei Deng
8. Song Han

Citations in the corpus

• Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, neutral): “Pruning [5] and quantization [9] methods share the same goal to improve the efficiency of ConvNets.”
• Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks (2019, background): “Some previous work [25, 4, 16, 3] has the “meta” word or some similar name such as “learning to prune”, but the core idea is different.”
• MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “Network pruning is an effective optimization, especially for constrained environments [13, 26, 55].”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “EAs have been introduced to compensate the loss error caused by the naive greedy pruning that in each iteration the top-k unimportant filters are heuristically pruned [213].”
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, unknown): “Pruning [24] and quantization [25] methods share the same goal to improve the efficiency of ConvNets.”