Improved Adam Optimizer for Deep Neural Networks
By Zijun Zhang. International Workshop on Quality of Service, 2018
1,333 citations recorded by the source corpus.
Adaptive optimization algorithms, such as Adam and RMSprop, have witnessed better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/46ffa583ba475ba699e5b1423e645a3c5dc9a0da
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Timothy P. Lillicrap
- Sergey Levine
- Adam Santoro
- Yoshua Bengio
- Greg Wayne
- Animesh Garg
- Pieter Abbeel
- Chelsea Finn
In Computer Vision and Pattern Recognition
- Adam Santoro
- Timothy P. Lillicrap
- Greg Wayne
- Luc Van Gool
- Li Fei-Fei
- Animesh Garg
- Dacheng Tao
- Trevor Darrell
Citations in the corpus
- Intra-agent speech permits zero-shot task acquisition (2022, methodology): “The loss was optimized by the Adam optimizer [27] with β 1 = 0 .”
- A modified Adam algorithm for deep neural network optimization (2023, methodology): “The modified algorithm, HN_Adam, is tested by using it to train a deep convolutional neural network using two different datasets CIFAR-10 [30] and MNIST [13].”
- Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “We used the ADAMax [95] optimizer to update the the hand pose parameters”