Deep Residual Networks and Weight Initialization

By Masato Taki. 2017

27 citations recorded by the source corpus.

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soham De
2. Samuel L. Smith
3. Yoshua Bengio
4. Quoc V. Le
5. Andrew Brock
6. Asja Fischer
7. Amos J. Storkey
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Soham De
4. Jitendra Malik
5. Cordelia Schmid
6. Trevor Darrell
7. Samuel L. Smith
8. Ming-Hsuan Yang

Citations in the corpus

• What do we need to build explainable AI systems for the medical domain? (2017, methodology): “Here, we are motivated by the fact that ResNet models have provided excellent classification results through the use of convolutional neural networks (Jegou et al., 2012; Xie et al., 2016; Huang and Peng, 2017; Taki, 2017).”
• How to Start Training: The Effect of Initialization and Architecture (2018, methodology): “The article [29] gives heuristic computations related to the mean squared activation in a depth-L ResNet and suggests taking the scales ⌘` of the residual modules to all be equal to 1/L (see §3.”
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (2020, background): “This is because, when the forward signal on the ` -th residual branch is downscaled by a factor ↵ , the backward propagated signal through that branch will also be downscaled by a factor ↵ [20].”
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2018, background): “, 2018) and use initialization scaled by the depth of the network (Taki, 2017).”
• Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “These insights from studies of batch-normalized ResNets are also supported by theoretical analyses of unnormalized networks (Taki, 2017; Yang & Schoenholz, 2017; Hanin & Rolnick, 2018; Qi et al., 2020).”