Normalized Direction-preserving Adam
By Zijun Zhang, Lin Ma, Zongpeng Li, Chuan Wu. 2017
31 citations recorded by the source corpus.
Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Boris Ginsburg
- Frank Hutter
- Richard Socher
- Nitish Shirish Keskar
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Alan L. Yuille
In Optimization & Training
- Richard Socher
- Nitish Shirish Keskar
- Frank Hutter
- Quanquan Gu
- Boris Ginsburg
- Quoc V. Le
- Ilya Loshchilov
- Yuan Cao
Citations in the corpus
- Fixing Weight Decay Regularization in Adam (2017, methodology): “It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017).”
- Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “Then the second-order moment v lt is computed for each layer l similar to ND-Adam [28]: The second-order moment v lt is used to re-scale gradients g lt before calculating the first-order moment m lt : If L2-regularization is used, a weight decay d · w t − 1 is added to the re-scaled gradient (as in…”
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, background): “Zhang et al. (2017) proposed a normalized direction-preserving Adam (ND-Adam), which changes the adaptive terms from individual dimensions to the whole gradient vector.”
- Improving Generalization Performance by Switching from Adam to SGD (2017, methodology): “Finally, we note that the strategy proposed in this paper does not preclude the use of those proposed in Zhang et al. (2017); Loshchilov & Hutter (2017); Anony-mous (2018).”
- A Survey of Optimization Methods From a Machine Learning Perspective (2019, background): “Recently some researchers are trying to explain and improve the adaptive methods [172], [173].”
- Decoupled Weight Decay Regularization (2017, methodology): “It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017).”