The Shattered Gradients Problem: If resnets are the answer, then what is the question?

By David Balduzzi, Marcus Frean, Lennox Leary, John P. Lewis, Kurt Wan-Duo Ma, Brian McWilliams. 2017

465 citations recorded by the source corpus.

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…wide models (Raghu et al., 2017; Arora et al., 2018), trainability challenges including exploding and vanishing gradients (Glo-rot and Bengio, 2010; Balduzzi et al., 2017) and empirical approaches to counter these challenges (Bapna et al., 2018; Zhang et al., 2019) are critical to further scaling…”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “This biases the signal towards the skip path, which ensures that the network has well-behaved gradients early in training, enabling efficient optimization (Balduzzi et al., 2017; Hanin & Rolnick, 2018; Yang et al., 2019).”
Fixup Initialization: Residual Learning Without Normalization (2019, methodology): “For residual networks, activation scale (Hanin & Rolnick, 2018), gradient variance (Balduzzi et al., 2017) and dynamical isometry property (Yang & Schoenholz, 2017) have been studied.”
On the Variance of the Adaptive Learning Rate and Beyond (2019, background): “…al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen et al., 2018), or adding range constraints (Luo et al., 2019)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba et al., 2016; Ioffe & Szegedy, 2015).”
Visualizing the Loss Landscape of Neural Nets (2017, background): “In this case, the gradients “shatter” [1], and training is impossible.”
Methods for interpreting and understanding deep neural networks (2017, background): “More specifically, they are subject to the problem of shattered gradients [7,49] occurring in deep ReLU networks: The number of piecewise-linear regions tends to grow very Fig.”
Opening the Black Box of Deep Neural Networks via Information (2017, background): “Another fundamental issue in machine learning, which we only deal with briefly in this paper, is the dependence on the training data size [Cho et al. (2015)].”
How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) (2018, background): “[2] observe that models without BatchNorm tend to suffer from small correlation between different gradient coordinates and/or unit activations.”
Learning Deep ResNet Blocks Sequentially using Boosting Theory (2017, background): “There are extensive works (Girshick, 2015; Rubinstein & Kroese, 2013; Tygert et al. , 2015) on selecting or modifying loss functions to prevent empirical difficulties such as exploding/vanishing gradients or slow learning (Balduzzi et al. , 2017).”
A Mean Field Theory of Batch Normalization (2019, positive): “(Balduzzi et al., 2017) found that batchnorm in resnets allow deep gradient signal propagation in contrast to the case without batchnorm.”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “Crucially, recent theoretical work (Balduzzi et al., 2017; De & Smith, 2020) has demonstrated that BatchNorm ensures good signal propagation at initialization in deep residual networks with identity skip connections (He et al.”
Interfacing photonics with artificial intelligence: an innovative design strategy for photonic structures and devices based on artificial neural networks (2021, background): “The residual connections and shortcut connections in the network are known as the residual learning [82] and U-Net [83] blocks, which can help to stabilize the gradient of the networks and make the network deeper without compromising its performance [84,85].”
MetaInit: Initializing learning by learning to initialize (2019, methodology): “CIFAR We use the β t -Swish ( x ) = σ ( β t x ) x activation function [49], with β 0 initialized to 0 to make the activation linear at initialization [5].”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “fer Learning 29 using backpropagation is an open problem [4]. Further work can be done to address these issues. Another potential problem for neural network training was discovered by Balduzzi et al. [7]. This is the “shattered gradient” problem, which is why residual networks with skip connections ”
NormFormer: Improved Transformer Pretraining with Extra Normalization (2021, methodology): “Similarly, some other approaches targeted initialization as well, in particular ReZero (Bachlechner et al., 2020), FixUp (Huang et al., 2020) and LookLinear (Balduzzi et al., 2017).”