Structured Pruning of Deep Convolutional Neural Networks

By Sajid Anwar, Kyuyeon Hwang, Wonyong Sung. 2015

824 citations recorded by the source corpus.

Real-time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks: feature map-wise, kernel-wise, and intra-kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, in parallel computing environments, and in hardware-based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by assessing the misclassification rate with a corresponding connectivity pattern. The pruned network is retrained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra-kernel strided sparsity with a simple constraint can significantly reduce the size of the kernel and feature map tensors. The proposed work shows that when pruning granularities are applied in combination, we can prune the CIFAR-10 network by more than 70% with less than a 1% loss in accuracy.

Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Song Han
  2. Yoshua Bengio
  3. Yi Yang
  4. Mohit Bansal
  5. Quoc V. Le
  6. Dacheng Tao
  7. Luke S. Zettlemoyer
  8. Jianfeng Gao

In Computational Mathematics

  1. Gaurav Menghani
  2. Suyog Gupta
  3. Michael Zhu
  4. Mark Anderson
  5. Sidak Pal Singh
  6. Carlos Gómez-Rodríguez
  7. Martin Jaggi
  8. Ari S. Morcos

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Yi Yang
  4. Trevor Darrell
  5. Ming-Hsuan Yang
  6. Xiaogang Wang
  7. Alan L. Yuille
  8. Jitendra Malik

In Model Architectures

  1. Jiahui Yu
  2. Thomas S. Huang
  3. Martin Jaggi
  4. Gaurav Menghani
  5. Sidak Pal Singh
  6. Kumar Shridhar
  7. Felix Laumann
  8. Suyog Gupta

In Efficiency & Compression

  1. Song Han
  2. Gaurav Menghani
  3. Jiahui Yu
  4. Thomas S. Huang
  5. Yang He
  6. Ji Lin
  7. Yi Yang
  8. Dan Alistarh

Citations in the corpus