Weighted residuals for very deep networks

By Falong Shen, Gang Zeng. 2016

53 citations recorded by the source corpus.

Deep residual networks have recently shown appealing performance on many challenging computer vision tasks. However, the original residual structure still has some defects making it difficult to converge on very deep networks. In this paper, we introduce a weighted residual network to address the incompatibility between ReLU and element-wise addition and the deep network initialization problem. The weighted residual network is able to learn to combine residuals from different layers effectively and efficiently. The proposed models enjoy a consistent improvement over accuracy and convergence with increasing depths from 100+ layers to 1000+ layers. Besides, the weighted residual networks have little more computation and GPU memory burden than the original residual networks. The networks are optimized by projected stochastic gradient descent. Experiments on CIFAR-10 have shown that our algorithm has a faster convergence speed than the original residual networks and reaches a high accuracy at 95.3% with a 1192-layer model. Experiments on CIFAR-100 and ImageNet-1k have also verified the effectiveness of our proposed design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ankur Bapna
2. Orhan Firat
3. P. Heng
4. Yuan Cao
5. Qi-Yan Dou
6. Yonghui Wu
7. Jianfei Cai
8. Lequan Yu

In Computer Vision and Pattern Recognition

1. G. Wang
2. Jianfei Cai
3. Luc Van Gool
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Trevor Darrell
7. Xiaogang Wang
8. Alan L. Yuille

Citations in the corpus

• Recent advances in convolutional neural networks (2015, background): “[148] introduce a weighting factor for the output from”
• Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “Similar approaches have been found to be useful in deep convolutional networks, for example (Shen and Zeng, 2016; Huang et al., 2016a; Srivastava et al., 2015; Huang et al., 2016b), but this remains un-investigated in sequence-to-sequence models.”
• Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “A prior work that also adds scaling factors for residual in ResNet is Weighted Residual Networks [36].”
• Residual Networks of Residual Networks: Multilevel Residual Networks (2016, methodology): “Inspired by Pre-ResNets, Shen and Zeng [33] proposed weighted residuals for very deep networks, which removed the ReLU from highway and used weighted residual functions to create a direct20?path.”
• VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images (2017, unknown): “Although 2D deep residual networks have been employed in many 2D natural image processing tasks (He et al., 2016a, 2016b; Shen et al., 2016; Zagoruyko and Komodakis, 2016; Lequan et al., 2016), to the best of our knowledge, seldom studies of residual learning have been dedicated to processing…”
• Deep Pyramidal Residual Networks (2016, methodology): “According to the work of Veit et al. [33], ResNets can be viewed as ensembles of relatively shallow networks, supported by the observation that deleting an individual building block in a residual unit of ResNets incurs minor classification loss, whereas removing layers from plain networks such as VG”
• VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation (2016, background): “Although 2D deep residual networks have been extensively studied in the domain of computer vision [10, 11, 29, 39], to the best of our knowledge, seldom studies have been explored in the field of medical image computing, where the majority of dataset is volumetric images.”