Understanding the difficulty of training deep feedforward neural networks

By Xavier Glorot, Yoshua Bengio. 2010

19,426 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning.”
Convolutional Sequence to Sequence Learning (2017, methodology): “For layers which are followed by a GLU activation, we pro-pose a weight initialization scheme by adapting the derivations in (He et al., 2015b; Glorot & Bengio, 2010; Appendix A).”
Recurrent Neural Network Grammars (2016, methodology): “All parameters were initialized according to recommendations given by Glorot and Bengio (2010).”
Training Very Deep Networks (2015, methodology): “All convolutional highway networks utilize the rectified linear activation function [16] to compute the block state H .”
Highway Networks (2015, methodology): “For plain deep networks, training with SGD stalls at the beginning unless a speciﬁc weight initialization scheme is used such that the variance of the signals during forward and backward propagation is preserved initially (Glorot & Bengio, 2010; He et al., 2015).”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “If a = 0, it becomes the ReLU case; if a = 1, it becomes the linear case (the same as [8]).”
Convolutional neural networks at constrained time cost (2014, methodology): “The weights are randomly initialized from zero-mean Gaussian distributions [7].”
Deeply-Supervised Nets (2014, background): “In addition to the present observation that learned features in CNN are not always intuitive and discriminative [29], broader discussion of difficulties in training deep neural networks has appeared in [8, 19].”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, methodology): “Indeed, many explanations for the difficulty of deep learning have been advanced in the literature, including the presence of many local minima, low curvature regions due to saturating nonlinearities, and exponential growth or decay of back-propagated gradients [12, 13, 14, 15].”
End-to-End Object Detection with Transformers (2020, methodology): “All transformer weights are initialized with Xavier init [11], and the backbone is with ImageNet-pretrained ResNet model [15] from torchvision with frozen batchnorm layers.”
Semi-Supervised Classification with Graph Convolutional Networks (2016, methodology): “As an example, consider the following 3-layer GCN model: with weight matrices W ( l ) initialized at random using the initialization described in Glorot & Bengio (2010).”
Graph Attention Networks (2017, methodology): “Both models are initialized using Glorot initialization (Glorot & Bengio, 2010) and trained to minimize cross-entropy on the training nodes using the Adam SGD optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.01 for Pubmed, and 0.005 for all other datasets.”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “The hidden-to-hidden weight matrix uses an orthogonal initialization (Saxe et al., 2013), and all other weight matrices use Xavier initialization (Glorot & Bengio, 2010).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “, 2018), trainability challenges including exploding and vanishing gradients (Glorot and Bengio, 2010; Balduzzi et al., 2017) and empirical approaches to counter these challenges (Bapna et al.”