Adam: A Method for Stochastic Optimization
By Diederik P. Kingma, Jimmy Ba. International Conference on Learning Representations, 2014
168,773 citations recorded by the source corpus.
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training
View Paper: https://arxiv.org/abs/1412.6980
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Kyunghyun Cho
In Management Science and Operations Research
- Sergey Levine
- Pieter Abbeel
- R´emi Munos
- Shimon Whiteson
- Frank Hutter
- Ofir Nachum
- Lihong Li
- Chelsea Finn
In Optimization & Training
- Yoshua Bengio
- Peter Richtárik
- Simon Lacoste-Julien
- Quoc V. Le
- Martin Jaggi
- Sebastian U. Stich
- Jakub Konecný
- Frank Hutter
Citations in the corpus
- Attention is All you Need (2017, methodology): “3 Optimizer We used the Adam optimizer [17] with β1 = 0.”
- A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “We train all our models with Adam (Kingma and Ba, 2014) with a batch size of 50 and a learning rate α of 0.001 for ML training and 0.0001 for RL and ML+RL training.”
- Structured Attention Networks (2017, methodology): “In the pretrained scenario, pretraining is done with Adam (Kingma & Ba, 2015) with learning rate equal to 0 .”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “We used the Adam optimizer (Kingma & Ba, 2015).”
- Neural Machine Translation in Linear Time (2016, methodology): “For the optimization we use Adam (Kingma and Ba, 2014) with a learning rate of 10−2 and a weight decay term of 10−5.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “We run Adam for the first 60k steps, after which we switch to simple SGD.”
- Layer Normalization (2016, methodology): “The model is trained using mini-batches of size 8 and the Adam [Kingma and Ba, 2014] optimizer.”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “We use Adam optimizer with a learning rate of 0.002 (Kingma and Ba 2014).”
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “For optimization, we use Adam (Kingma and Ba, 2015) with a learning rate of 10 − 3 and clip gradients to norm 1 .”
- Neural Network Acceptability Judgments (2018, methodology): “All neural network models are implemented in PyTorch and optimized using Adam (Kingma and Ba, 2014).”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “We train our models with Adam (Kingma and Ba, 2014) with initial learning rate 10 − 3 and batch size 128.”
- Deep Contextualized Word Representations (2018, methodology): “Parameters were optimized using Adam (Kingma and Ba, 2015) with gradient norms clipped at 5.0 and initial learning rate 0.0004, decreasing by half each time accuracy on the development set did not increase in subsequent epochs.”
- MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “A T RAINING D ETAILS Our model was trained with the Adam method for stochastic optimization (Kingma & Ba, 2015) with the default Tensorflow exponential decay rates of β 1 = 0 .”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “For a given model, the transfer quality is also sensitive to the optimization algorithm: when training with Adam instead of SGD, we observed that the BiLSTM-max converged faster on SNLI (5 epochs instead of 10), but obtained worse re-sults on the transfer tasks, most likely because of the model and ”