Network Decoupling: From Regular to Depthwise Separable Convolutions

By Jianbo Guo, Yuxi Li, Weiyao Lin, Yurong Chen, Jianguo Li. 2018

100 citations recorded by the source corpus.

Depthwise separable convolution has shown great efficiency in network design, but requires time-consuming training procedure with full training-set available. This paper first analyzes the mathematical relationship between regular convolutions and depthwise separable convolutions, and proves that the former one could be approximated with the latter one in closed form. We show depthwise separable convolutions are principal components of regular convolutions. And then we propose network decoupling (ND), a training-free method to accelerate convolutional neural networks (CNNs) by transferring pre-trained CNN models into the MobileNet-like depthwise separable convolution structure, with a promising speedup yet negligible accuracy loss. We further verify through experiments that the proposed method is orthogonal to other training-free methods like channel decomposition, spatial decomposition, etc. Combining the proposed method with them will bring even larger CNN speedup. For instance, ND itself achieves about 2X speedup for the widely used VGG16, and combined with other methods, it reaches 3.7X speedup with graceful accuracy degradation. We demonstrate that ND is widely applicable to classification networks like ResNet, and object detection network like SSD300.

Subfields: Computational Mechanics, Computer Networks and Communications, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Liang Wang
2. Jun Du
3. Changshui Zhang
4. Gregory Hager
5. Varun Jain
6. Zhuang Liu
7. Chen Gao
8. Jianguo Li

In Computer Networks and Communications

1. Liang Wang
2. Jun Du
3. Changshui Zhang
4. Zhuang Liu

In Signal Processing

1. Liang Wang
2. Jun Du
3. Changshui Zhang
4. Gregory Hager
5. Zhuang Liu
6. Zirui Wang
7. Jianguo Li
8. Tianhong Li

Citations in the corpus

• sharpDARTS: Faster and More Accurate Differentiable Architecture Search (2019, background): “[8, 3] Furthermore, so-called “bot-tleneck” layers or blocks have proven useful to limiting the size and improving the accuracy of neural network models[10, 23].”
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search (2020, methodology): “Our search space of operations for feature transformation includes: 2D convolution, add every N consecutive channels [77, 78], separable convolution [28, 77, 78], depth-wise convolution [23,28,29], and identity.”
• Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, unknown): “2016), and depth-wise (Guo et al. 2018) decompositions, can be adopted.”
• Knowledge Distillation from Few Samples (2018, methodology): “For the former case, we evaluate FSKD on three well-known compression methods, filter pruning (Li et al., 2016), network slimming (Liu et al., 2017), and network decoupling (Guo et al., 2018).”
• Supplementary Material for Few Sample Knowledge Distillation for Efficient Network Compression (2020, methodology): “Please refer to [1] for the details of proof for this Lemma.”
• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, background): “convolution may not always hurt the model performance, since separable convolution has been proved to capture the most effective part of standard convolutions and meanwhile discard other redundant parts [45].”
• Joint architecture and knowledge distillation in CNN for Chinese text recognition (2019, background): “In addition, Guo et al. [56] proved that DSConv is the principal components of standard convolution and can approximate the standard convolution in closed form.”