Random Walk Initialization for Training Very Deep Feedforward Networks

By David Sussillo, L. F. Abbott. 2014

82 citations recorded by the source corpus.

Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent network, application to an FFN amounts to multiplying the error gradient by a different random matrix at each layer. We show that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors, and we compute the scaling that makes this walk unbiased. The variance of the random walk grows only linearly with network depth and is inversely proportional to the size of each layer. Practically, this implies a gradient whose log-norm scales with the square root of the network depth and shows that the vanishing gradient problem can be mitigated by increasing the width of the layers. Mathematical analyses and experimental results using stochastic gradient descent to optimize tasks related to the MNIST and TIMIT datasets are provided to support these claims. Equations for the optimal matrix scaling are provided for the linear and ReLU cases.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Training Very Deep Networks (2015, neutral): “Other similarly motivated strategies have shown promising results in preliminary experiments [18, 19].”
Data-dependent Initializations of Convolutional Neural Networks (2015, positive): “Glorot & Bengio (2010); Saxe et al. (2013); Sussillo & Abbot (2015); He et al. (2015); Bradley (2010) show that properly scaled random initialization can deal with the vanishing gradient problem, if the architectures are limited to linear transformations, followed by a very specific non-linearities.”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, methodology): “The observation that the norms of weights form a random walk was used by Sussillo & Abbott (2015) to tune the gains of neurons.”
Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both? (2019, background): “In [134], it is shown that, by properly choosing the gain factor, the orthogonality assumption of the weight matrices can be relaxed.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, background): “Saxe, McClelland, and Ganguli (2013); Sussillo and Abbott (2014), Hinton, Vinyals, and Dean (2015), Romero et al. (2015), and Srivastava (2015a, 2015b) can be referred to for other appropriate techniques.”
Gradual DropIn of Layers to Train Very Deep Neural Networks (2015, methodology): “Sussillo and Abbott [23] suggest an initialization scheme called Random Walk Initialization based on scaling the initial random matrices correctly.”
Long short-term memory and Learning-to-learn in networks of spiking neurons (2018, methodology): “In order to optimize not only the synaptic weights of a RSNN but also its connectivity matrix, we integrated BPTT with the biologically inspired [3] rewiring method DEEP R [4] (see Suppl.”
A Review of Deep Learning with Special Emphasis on Architectures, Applications and Recent Trends (2019, methodology): “Lately, many other techniques like l1, l2 regularisation [42], dropout [43], batch normalisation [44], good set of weight initialisation techniques [45], [46], [47], Fig.”
Efficient Summarization with Read-Again and Copy Mechanism (2016, background): “All parameters, except the biases are initialized uniformly with a range of 3 /d , where d is the dimension of the hidden state (Sussillo & Abbott (2014)).”