On the Convergence of Adam and Beyond
By Sashank J. Reddi, Satyen Kale, Surinder Kumar. 2018
2,997 citations recorded by the source corpus.
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.
Subfields: Artificial Intelligence, Numerical Analysis, Optimization & Training, Philosophy
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Sergey Levine
- Ruslan Salakhutdinov
In Numerical Analysis
- Simon Lacoste-Julien
- Nicolas Loizou
- Jakub Konecný
- Sebastian Pokutta
- Francesco Orabona
- Sebastian U. Stich
- Mikhail Belkin
- Issam Hadj Laradji
In Optimization & Training
- Simon Lacoste-Julien
- Peter Richtárik
- Martin Jaggi
- Nicolas Loizou
- Sebastian U. Stich
- Francis R. Bach
- Jakub Konecný
- Quoc V. Le
In Philosophy
- Mikhail Belkin
- Chaoyue Liu
- Jean-Marc Valin
- Diganta Misra
- Jan Skoglund
- Francesco Orabona
- Yang Song
- Stefano Ermon
Citations in the corpus
- Memory-Efficient Adaptive Optimization for Large-Scale Learning (2019, methodology): “Several variants of Adagrad have now been proposed in the optimization and machine learning literature (see [19] and the references therein), the most notable of which is Adam [15].”
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “For all other experiments, we use AMSGrad (Reddi et al., 2018).”
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost (2018, methodology): “An alternative solution to the problems described in Section 5 is to use an increasing schedule of β 2 , as proposed by Reddi et al. (2018).”
- On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Others do not even report the value of (cid:15) used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019).”
- Improved Techniques for Training Score-Based Generative Models (2020, methodology): “We note, however, this is a well-known problem of the Adam optimizer, and can be mitigated by techniques such as AMSGrad [22].”
- Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution (2022, methodology): “In Figures 5&12&13 we compare DecSPS with Adam [19] and AMSgrad [26] on the A1A and Breast Cancer datasets.”
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
- Fixing Weight Decay Regularization in Adam (2017, neutral): “, 2011), RMSProp (Tieleman & Hinton, 2012), and AMSGrad (Reddi et al., 2018).”
- Stable and low-precision training for large-scale vision-language models (2023, background): “On the other hand, if β2 is reduced too much then learning is slowed which results in worse performance [48].”
- On the Variance of the Adaptive Learning Rate and Beyond (2019, background): “It describes various popular stochastic gradient descent algorithms (Reddi et al., 2018).”
- Adaptive Federated Optimization (2020, methodology): “…learning community because they can reduce the need to tune the learning rates and often exhibit strong empirical performance especially in presence of heavy-tail noise distributions of stochastic gradients (Reddi et al., 2019; McMahan and Streeter, 2010; Duchi et al., 2011; Zhang et al., 2019).”
- QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, methodology): “We design a new fixed-point optimization algorithm inspired by a state-of-the-art floating-point adaptive gradient optimization procedure [47].”
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “SIGN SGD and ADAM are empirically known to generalize worse than SGD Wilson et al. [2017], Balles and Hennig [2018].”
- Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms (2019, background): “[20] that Adam [14] can diverge in some cases.”
- Federated Learning with Matched Averaging (2020, methodology): “To make the comparison fair, we use AMSGRAD for all other centralized baselines to get the reported results in our experiments.”