Extreme Network Compression via Filter Group Approximation

By Bo Peng, Wenming Tan, Zheyang Li, Shun Zhang, Di Xie, Shiliang Pu. 2018

67 citations recorded by the source corpus.

In this paper we propose a novel decomposition method based on filter group approximation, which can significantly reduce the redundancy of deep convolutional neural networks (CNNs) while maintaining the majority of feature representation. Unlike other low-rank decomposition algorithms which operate on spatial or channel dimension of filters, our proposed method mainly focuses on exploiting the filter group structure for each layer. For several commonly used CNN models, including VGG and ResNet, our method can reduce over 80% floating-point operations (FLOPs) with less accuracy drop than state-of-the-art methods on various image classification datasets. Besides, experiments demonstrate that our method is conducive to alleviating degeneracy of the compressed network, which hurts the convergence and performance of the network.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luc Van Gool
2. Greg Mori
3. Dengxin Dai
4. Jonathan Frankle
5. Frederick Tung
6. Suman Saha
7. John V. Guttag
8. Davis W. Blalock

In Computational Mechanics

1. Greg Mori
2. Luc Van Gool
3. John V. Guttag
4. Dengxin Dai
5. Frederick Tung
6. Stamatios Georgoulis
7. Radu Timofte
8. Chen Change Loy

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dengxin Dai
3. Greg Mori
4. Ming-Hsuan Yang
5. Cordelia Schmid
6. Xiaogang Wang
7. Dacheng Tao
8. Jitendra Malik

In Efficiency & Compression

1. Jonathan Frankle
2. Greg Mori
3. John V. Guttag
4. Davis W. Blalock
5. Jose Javier Gonzalez Ortiz
6. Wayne W. C. Luk
7. Luc Van Gool
8. Luca Zappella

Citations in the corpus

• What is the State of Neural Network Pruning? (2020, background): “…other metrics, such as power consumption (Louizos et al., 2017; Yang et al., 2017; Han et al., 2015; Kim et al., 2015), memory bandwidth usage (Peng et al., 2018; Kim et al., 2015), or fine-tuning time (Dubey et al., 2018; Yamamoto & Maeno, 2018; Huang & Wang, 2018; He et al., 2018a), and we…”
• Similarity-Preserving Knowledge Distillation (2019, methodology): “Lowrank factorization methods [6, 7, 16, 27, 45] produce compact low-rank approximations of filter matrices.”
• Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “For the two configurations from [32], ResNet-34/A and /B, models produced by us are smaller but less accurate.”
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “In particular, filter-wise decomposition into a product of low-rank filters [20], filter groups [47], a basis of filter groups [30], etc.”
• T-Basis: a Compact Representation for Neural Networks (2020, background): “However, Peng et al. (2018) showed that better compression could be achieved if we exploit the filter group structure of each layer.”
• Variational Convolutional Neural Network Pruning (2019, unknown): “These approaches include tensor factorization [35, 48], net-”
• NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “Specifically we will contrast PFA against the filter pruning approach (FP) in Li et al. (2017), the network slimming approach (NS) in Liu et al. (2017), the variational information bottleneck approach (VIB) in Dai et al. (2018) and the filter group approximation approach (FGA) in Peng et al. (2018).”