The Marginal Value of Adaptive Gradient Methods in Machine Learning

By Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, B. Recht. 2017

1,163 citations recorded by the source corpus.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “However while SGD ﬁnds minima that generalize well (Zhang et al., 2016; Wilson et al., 2017), each parameter update only takes a small step towards the objective.”
On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “The sole exceptions to this bootstrap procedure are the two left panels of Figure 3, for which we used a similar procedure to Wilson et al. (2017) to ensure comparability.”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, methodology): “This strategy has been first introduced for variance reduced methods by Xiao and Zhang (2014) for SVRG, and has been discussed for saddle point optimization by Palaniappan and Bach (2016).”
Fixing Weight Decay Regularization in Adam (2017, methodology): “Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016; Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).”
Regularizing and Optimizing LSTM Language Models (2017, positive): “This is in agreement with recent evidence pointing to the insufﬁciency of adaptive gradient methods ( Wilson et al., 2017).”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “, 2018) as well the generalization performance (Wilson et al., 2017) of such adaptive algorithms has been called into question.”
Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, methodology): “Moreover, popular approach is to use dedicated optimizers such as Momentum, AdaGrad, Adam [7], learning rate schedules [8], or techniques such as batch normalization [9], online batch selection.”
Invariant Risk Minimization (2019, background): “Amongst all the zero training error solutions, these procedures will prefer the solution with the smallest capacity [53, 3].”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “Wilson et al. [72] show that even for a toy quadratic problem, Adam generalizes provably worse than SGD.”
signSGD: compressed optimisation for non-convex problems (2018, background): “Wilson et al. (2017) observed that ADAM tends to generalise slightly worse than SGD.”
Incremental Learning Through Deep Adaptation (2017, background): “g, SGD), at the cost of possibly lower final accuracy [41].”
The Implicit Bias of Gradient Descent on Separable Data (2017, background): “Introduction It is becoming increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models (Neyshabur et al., 2014, 2015; Zhang et al., 2017; Keskar et al., 2017; Neyshabur et al., 2017; Wi”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Wilson et al. (2017) found SGD consistently generalized better to the test set than second-order methods.”
Adaptive Gradient Methods with Dynamic Bound of Learning Rate (2019, methodology): “A DAM in particular has become the default algorithm leveraged across many deep learning frameworks due to its rapid training speed (Wilson et al., 2017).”
Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods (2017, methodology): “which is being used extensively with mSGD for training deep neural networks [84,88,94].”
YellowFin and the Art of Momentum Tuning (2017, positive): “This increased ﬂexibility sounds great, however our experiments and recent analysis in literature [1] suggest that methods that adapt multiple learning rates, yield marginal beneﬁts compared to momentum SGD. Wilson et al. [1] argue that those methods also have worse generalization.”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “…we consider some optimization based measures based on the speed of the optimization algorithm as motivated by the work of (Hardt et al., 2015) and (Wilson et al., 2017a), and the magnitude of the gradient noise as motivated by the work of (Chaudhari and Soatto, 2018) and (Smith and Le, 2017).”
On the importance of single directions for generalization (2018, background): “Others have focused on the information content stored in network weights (Achille & Soatto, 2017), while still others have demonstrated that stochastic gradient descent itself encourages generalization (Bousquet & Elisseeff, 2002, Smith & Le, 2017, Wilson et al., 2017).”