Incorporating Nesterov Momentum into Adam

By Timothy Dozat. 2016

2,074 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “…standard to leave (cid:15) at the common default value of 10 − 8 for Adam and 10 − 10 for RMSProp (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al.,…”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “Frame-wise cross-entropy loss criterion and Adam optimization with Nesterov momentum (Nadam) are used for the mini-batch training of the network [15, 16].”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “, and others (Duchi et al., 2011; Dozat, 2016; Martens & Grosse, 2015)) has become an important design choice, though understanding of its relationship to model generalization remains nascent (Shirish Keskar et al., 2016; Wilson et al., 2017; Shirish Keskar & Socher, 2017; Agarwal et al., 2020;…”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Dozat (2016) incorporated Nesterov’s momentum into Adam optimizer and proposed the Nadam optimizer.”
On the Convergence of Adam and Beyond (2018, methodology): “We will mainly focus on A DAM algorithm due to this generality but our arguments also apply to RMS PROP and other algorithms such as A DADELTA , N ADAM .”
On the Variance of the Adaptive Learning Rate and Beyond (2019, background): “In particular, Adagrad (Duchi et al., 2010) and its variants, e.g. , RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2014), Adadelta (Zeiler, 2012) and Nadam (Dozat, 2016), stand out due to their fast convergence, and have been considered as the optimizer of choice in many applications.”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, background): “d the dropout mechanism have a high impact on the achieved performance. We showed that the embeddings byKomninos and Manandhar (2016) typically perform best, that Adam with Nesterov momentum (Nadam) (Dozat, 2015) results in the best performance and converges the fastest, that a gradient normalizatio”
Predictive Uncertainty Estimation via Prior Networks (2018, methodology): “Models were implemented in Tensorflow [28], trained using the NADAM optimizer [29] with dropout regularization [30] and ReLU activation functions with an exponentially decaying learning rate.”
An overview of gradient descent optimization algorithms (2016, methodology): “ponentially decaying average of past gradients m t. We have also seen that Nesterov accelerated gradient (NAG) is superior to vanilla momentum. Nadam (Nesterov-accelerated Adaptive Moment Estimation) [7] thus combines Adam and NAG. In order to incorporate NAG into Adam, we need to modify its momentu”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, methodology): “We train the model using the Nadam (Dozat, 2016) algorithm for adaptive stochastic gradient descent (SGD), which combines Adam (Kingma and Ba, 2015) SGD with Nesterov momentum (Nesterov, 1983).”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, background): “The optimizers RMSProp, Adam, and Nadam on the other hand produced much more stable results.”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, methodology): “Many variants of Adagrad, such as RMSprop (Hinton et al., 2012), Adam (Kingma and Ba, 2015), Adadelta (Zeiler, 2012), Nadam (Dozat, 2016), were then proposed to address these challenges by adopting exponential moving average rather than the arithmetic average used in Adagrad.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “r the cascaded graph partition module under different object sizes. The cascade scheme signiﬁcantly reduces the time for large objects. Best viewed in color. work [9]. Our model is trained with Nadam [16] for 70,000 iterations using synchronized batch normalization [22] over 8 TitanX 1080ti GPUs and”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, methodology): “We again used Nadam for model parameter optimization and Adam for pseudo example optimization, and both of the initial learning rates were set to 0.02.”
A Sufficient Condition for Convergences of Adam and RMSProp (2018, methodology): “Although RMSProp, Adam, and Nadam work well for solving large-scale convex and non-convex optimization problems such as training deep neural networks, they have been pointed out to be divergent in some scenarios via convex counterexamples [20].”
Mish: A Self Regularized Non-Monotonic Neural Activation Function (2019, methodology): “In Figure 21, it is observed Mish significantly outperforms Swish while using NADAM [18,19] optimizer.”
Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network (2018, methodology): “er a small random batch of image patches, followed by back-propagation of the error signal through the network). In detail, we use the Adam variant of SGD (Kingma and Ba,2014) with Nesterov momentum (Dozat, 2015). Empirically, the proposed network architecture converges faster than other ones we exp”
Solving parametric PDE problems with artificial neural networks (2017, positive): “A meansquared-error loss function is used and the optimisation is done using the NAdam optimiser [5].”