Doubly Convolutional Neural Networks

By Shuangfei Zhai, Yu Cheng, Zhongfei Zhang, Weining Lu. 2016

66 citations recorded by the source corpus.

Building large models with parameter sharing accounts for most of the success of deep convolutional neural networks (CNNs). In this paper, we propose doubly convolutional neural networks (DCNNs), which significantly improve the performance of CNNs by further exploring this idea. In stead of allocating a set of convolutional filters that are independently learned, a DCNN maintains groups of filters where filters within each group are translated versions of each other. Practically, a DCNN can be easily implemented by a two-step convolution procedure, which is supported by most modern deep learning libraries. We perform extensive experiments on three image classification benchmarks: CIFAR-10, CIFAR-100 and ImageNet, and show that DCNNs consistently outperform other competing architectures. We have also verified that replacing a convolutional layer with a doubly convolutional layer at any depth of a CNN can improve its performance. Moreover, various design choices of DCNNs are demonstrated, which shows that DCNN can serve the dual purpose of building more accurate models and/or reducing the memory footprint without sacrificing the accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Furu Wei
3. Jingjing Liu
4. Yu Cheng
5. Dacheng Tao
6. Jianfeng Gao
7. Tao Ge
8. Canwen Xu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Zhe Gan
3. Luc Van Gool
4. Jingjing Liu
5. Yu Cheng
6. Yi Yang
7. Ming-Hsuan Yang
8. Trevor Darrell

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “• Transferred compact convolutional filters: These meth-ods remove inessential parameters by transferring or compressing the convolutional filters (Zhai et al. 2016).”
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “Conventional explanations [3, 43] claim that the large number of weights is necessary for the training of deep neural network but a high degree of redundancy exists after training.”
• Coreset-Based Neural Network Compression (2018, methodology): “rotation) within the data, by modifying the filter banks of a CNN [61,10] or by enforcing sparsity while training [2].”
• Patient Knowledge Distillation for BERT Model Compression (2019, background): “A conventional understanding is that a large number of connections (weights) is necessary for training deep networks (Denil et al., 2013; Zhai et al., 2016).”
• Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “Baseline Models Representative Works Alexnet [1] structural matrix [30]–[32] low-rank factorization [39] Network in network [68] low-rank factorization [39] VGG nets [69] transfered filters [43] low-rank factorization [39] Residual networks [70] compact filters [48], stochastic depth [61] parameter ”
• A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, methodology): “layer with global average pooling [43], [60].”
• Analysis and Optimization of Convolutional Neural Network Architectures (2017, background): “tion,thismightbeanindicatorfortoo little capacity of that layer. Hence adding more filters to that layer could improve the performance. Filterscanbecomparedwiththek-translationcorrelationasintroducedin[ZCZL16]: ˆ k(W i;W j) = max (x;y)2f k;:::;kg2n(0;0) hW i;T(W j;x;y)i f kW ik 2 kW jk 2[ 1;1]; where”