The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs

By Yong Guo, Mingkui Tan, Qingyao Wu, Jian Chen, Anton van den Hengel, Javen Qinfeng Shi. 2016

13 citations recorded by the source corpus.

Depth is one of the key factors behind the success of convolutional neural networks (CNNs). Since ResNet, we are able to train very deep CNNs as the gradient vanishing issue has been largely addressed by the introduction of skip connections. However, we observe that, when the depth is very large, the intermediate layers (especially shallow layers) may fail to receive sufficient supervision from the loss due to the severe transformation through a long backpropagation path. As a result, the representation power of intermediate layers can be very weak and the model becomes very redundant with limited performance. In this paper, we first investigate the supervision vanishing issue in existing backpropagation (BP) methods. And then, we propose to address it via an effective method, called Multi-way BP (MW-BP), which relies on multiple auxiliary losses added to the intermediate layers of the network. The proposed MW-BP method can be applied to most deep architectures with slight modifications, such as ResNet and MobileNet. Our method often gives rise to much more compact models (denoted by "Mw+Architecture") than existing methods. For example, MwResNet-44 with 44 layers performs better than ResNet-110 with 110 layers on CIFAR-10 and CIFAR-100. More critically, the resultant models even outperform the light models obtained by state-of-the-art model compression methods. Last, our method inherently produces multiple compact models with different depths at the same time, which is helpful for model selection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingkui Tan
2. Jian Chen
3. Yong Guo
4. Qi Chen
5. Junzhou Huang
6. Jingdong Wang
7. Zhenhao He
8. Yongsheng Luo

In Computer Vision and Pattern Recognition

1. Mingkui Tan
2. Jian Chen
3. Yong Guo
4. Junzhou Huang
5. Qi Chen
6. Jingdong Wang
7. Zhenhao He
8. Yongsheng Luo

Citations in the corpus

• Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, background): “In practice, designing deep models is often very labor-intensive and relies heavily on human expertise [10]–[13].”
• Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “1 The details of the fine-tuning algorithm can be found in [16].”
• Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution (2020, methodology): “Deep neural networks (DNNs) have been the workhorse of many real-world applications, including image classification [18, 14, 9, 15, 27, 13], video understanding [46, 45,”