On the importance of initialization and momentum in deep learning

By Ilya Sutskever, James Martens, George E. Dahl, Geoffrey E. Hinton. 2013

5,312 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, background): “Decaying β 1 ,t towards one is important in our theoretical analysis and also matches previous emperical ﬁndings, e.g. (Sutskever et al., 2013) suggests reducing the momentum coefﬁcient in the end of training can improve convergence.”
Convolutional Sequence to Sequence Learning (2017, methodology): “We train our convolutional models with Nesterov’s accelerated gradient method (Sutskever et al., 2013) using a momentum value of 0 .”
Rethinking the Inception Architecture for Computer Vision (2015, methodology): “Our earlier experiments used momentum [19] with a decay of 0 .”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “The model is trained using SGD with momentum (Polyak, 1964; Sutskever et al., 2013).”
Training Very Deep Networks (2015, background): “To deal with the difﬁculties of training deep networks, some researchers have focused on developing better optimizers (e.g. [13, 14, 15]).”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “At the cost of expressivity, deep linear networks gain theoretical tractability and may prove fertile for addressing other phenomena in deep learning, such as the impact of carefully-scaled initializations [13, 23], momentum [23], dropout regularization [1], and sparsity constraints [2].”
An Empirical Model of Large-Batch Training (2018, methodology): “999 ), and CIFAR10 and ImageNet with momentum [SMDH13] (with momentum = 0 .”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “(2017) we use Nesterov’s accelerated gradient method (Sutskever et al., 2013) with a momentum of 0.”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “e iterates tfor which we estimate that L( t) is low using a validation data set. Variants of SGD commonly used with neural networks include SGD with momentum (Polyak, 1964; Rumelhart et al., 1986; Sutskever et al., 2013), Nesterov momentum (Nesterov, 1983; Sutskever et al., 2013), RMSProp (Hinton e”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “(2017) we use Nesterov’s accelerated gradient method (Sutskever et al., 2013) with a momentum value of 0.”
Word-Entity Duet Representations for Document Ranking (2017, methodology): “Training loss was optimized using the Nadam algorithm [18].”
Character-level Convolutional Networks for Text Classification (2015, methodology): “The algorithm used is stochastic gradient descent (SGD) with a minibatch of size 128, using momentum [26] [30] 0.”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “We trained our convolutional models with Nesterov’s accelerated gradient method (Sutskever et al., 2013) using a momentum value of 0.”
Hierarchical Neural Story Generation (2018, methodology): “Similar to Gehring et al. (2017), we train using the Nesterov accelerated gradient method (Sutskever et al., 2013) using gradient clipping (Pascanu et al., 2013).”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “We use stochastic gradient descent (SGD) optimizer with Nesterov momentum [34].”