Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?

By Boris Hanin. 2018

302 citations recorded by the source corpus.

We give a rigorous analysis of the statistical behavior of gradients in randomly initialized feed-forward networks with ReLU activations. Our results show that a fully connected depth $d$ ReLU net with hidden layer widths $n_j$ will have exploding and vanishing gradients if and only if $\sum_{j=1}^{d-1} 1/n_j$ is large. The point of view of this article is that whether a given neural net will have exploding/vanishing gradients is a function mainly of the architecture of the net, and hence can be tested at initialization. Our results imply that a fully connected network that produces manageable gradients at initialization must have many hidden layers that are about as wide as the network is deep. This work is related to the mean field theory approach to random neural nets. From this point of view, we give a rigorous computation of the $1/n_j$ corrections to the propagation of gradients at the so-called edge of chaos.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

Fixup Initialization: Residual Learning Without Normalization (2019, background): “Recently, the study of dynamical isometry (Saxe et al., 2013) provides a more detailed characterization of the forward and backward signal propogation at initialization (Pennington et al., 2017; Hanin, 2018), enabling training 10,000-layer CNNs from scratch (Xiao et al., 2018).”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “The article [7] studied the more delicate question of the variance for gradients computed by random ReLU nets.”
A Survey of Deep Learning for Scientific Discovery (2020, background): “…often tricky to develop and train, due to their recursiveness presenting challenges with optimization (the vanishing/exploding gradients problem [87, 170, 76]), with performing fast model training (due to generating targets token by token), and challenges learning long term sequential dependencies.”
Random Walks and Heat Kernels on Graphs (2017, background): “Recently, applications of random walks on graphs were found in the study of deep neural nets (see [7] for some background).”
Adversarial Initialization - when your network performs the way I want (2019, background): “State of the art approaches rely on the idea that given a random initialization, the variance of weights is particularly important [11], [12] and determines the dynamics of the networks [16], [30].”
A Fine-Grained Spectral Perspective on Neural Networks (2019, background): “Closely related to the discussion of CK and NTK is the signal propagation literature, which tries to understand how to prevent pathological behaviors in randomly initialized neural networks when they are deep [48, 52, 71, 70, 23, 24, 14, 72, 44, 25, 47].”
Finite Depth and Width Corrections to the Neural Tangent Kernel (2019, methodology): “Also, the sum-over-path approach to studying correlation functions in randomly initialized ReLU nets was previously taken up for the foward pass in [11] and for the backward pass in [9] and [10].”