Learning Instance-wise Sparsity for Accelerating Deep Models

By Chuanjian Liu, Yunhe Wang, Kai Han, Chunjing Xu. 2019

30 citations recorded by the source corpus.

Exploring deep convolutional neural networks of high efficiency and low memory usage is very essential for a wide variety of machine learning tasks. Most of existing approaches used to accelerate deep models by manipulating parameters or filters without data, e.g., pruning and decomposition. In contrast, we study this problem from a different perspective by respecting the difference between data. An instance-wise feature pruning is developed by identifying informative features for different instances. Specifically, by investigating a feature decay regularization, we expect intermediate feature maps of each instance in deep neural networks to be sparse while preserving the overall network performance. During online inference, subtle features of input images extracted by intermediate layers of a well-trained neural network can be eliminated to accelerate the subsequent calculations. We further take coefficient of variation as a measure to select the layers that are appropriate for acceleration. Extensive experiments conducted on benchmark datasets and networks demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Yunhe Wang
3. Yulin Wang
4. Kai Han
5. Gao Huang
6. Chunjing Xu
7. Jianyuan Guo
8. Qi Tian

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Kai Han
3. Shiji Song
4. Gao Huang
5. Jianyuan Guo
6. Chunjing Xu
7. Qi Tian
8. Yulin Wang

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Yunhe Wang
7. Honghui Wang
8. Kai Han

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “c) Dynamic pruning based on feature activations directly has also been realized without auxiliary branches and computational overheads [91], where a regularization item is”
• GhostNet: More Features From Cheap Operations (2019, background): “Channel pruning [51, 18, 31, 39, 59, 23, 35] further targets on removing useless channels for easier acceleration in practice.”