Design of Efficient Convolutional Layers using Single Intra-channel Convolution, Topological Subdivisioning and Spatial "Bottleneck" Structure

By Min Wang, Baoyuan Liu, Hassan Foroosh. 2016

53 citations recorded by the source corpus.

Deep convolutional neural networks achieve remarkable visual recognition performance, at the cost of high computational complexity. In this paper, we have a new design of efficient convolutional layers based on three schemes. The 3D convolution operation in a convolutional layer can be considered as performing spatial convolution in each channel and linear projection across channels simultaneously. By unravelling them and arranging the spatial convolution sequentially, the proposed layer is composed of a single intra-channel convolution, of which the computation is negligible, and a linear channel projection. A topological subdivisioning is adopted to reduce the connection between the input channels and output channels. Additionally, we also introduce a spatial "bottleneck" structure that utilizes a convolution-projection-deconvolution pipeline to take advantage of the correlation between adjacent pixels in the input. Our experiments demonstrate that the proposed layers remarkably outperform the standard convolutional layers with regard to accuracy/complexity ratio. Our models achieve similar accuracy to VGG, ResNet-50, ResNet-101 while requiring 42, 4.5, 6.5 times less computation respectively.

Subfields: Aerospace Engineering, Computer Networks and Communications, Electrical and Electronic Engineering, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Hartwig Adam
2. Yukun Zhu
3. Liang-Chieh Chen
4. Jonathon Shlens
5. Florian Schroff
6. Barret Zoph
7. Maxwell D. Collins
8. George Papandreou

In Computer Networks and Communications

1. Amos J. Storkey
2. Yukun Zhu
3. Barret Zoph
4. Hartwig Adam
5. George Papandreou
6. Maxwell D. Collins
7. Florian Schroff
8. Liang-Chieh Chen

In Electrical and Electronic Engineering

1. Gavia Gray
2. Amos J. Storkey
3. Jonathon Shlens
4. Yukun Zhu
5. Barret Zoph
6. Hartwig Adam
7. George Papandreou
8. Florian Schroff

In Model Architectures

1. Liang-Chieh Chen
2. Yukun Zhu
3. Hartwig Adam
4. Amos J. Storkey
5. Jonathon Shlens
6. Barret Zoph
7. Quoc V. Le
8. Florian Schroff

In Efficiency & Compression

1. Jonathon Shlens
2. Liang-Chieh Chen
3. Barret Zoph
4. Yukun Zhu
5. Hartwig Adam
6. Gavia Gray
7. Amos J. Storkey
8. George Papandreou

Citations in the corpus

• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, methodology): “This operation has been adopted in many recent neural network designs [35, 74, 12, 31, 80, 60, 84].”
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “We employ separable convolution [79, 85, 86, 17, 38] with 256 filters for all the convolutions, and decouple sampling rates in the 3× 3 atrous separable convolution to be rh × rw which allows us to capture object scales with different aspect ratios.”
• Moonshine: Distilling with Cheap Convolutions (2017, background): “Wang et al. (2016) start with separable convolutions and add topological subdivisioning, a way to treat sections of tensors separately, and a bottleneck of the spatial dimensions.”