Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

By Lukas Balles, Philipp Hennig. 2017

229 citations recorded by the source corpus.

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an estimate of their relative variance. We disentangle these two aspects and analyze them in isolation, gaining insight into the mechanisms underlying ADAM. This analysis also extends recent results on adverse effects of ADAM on generalization, isolating the sign aspect as the problematic one. Transferring the variance adaptation to SGD gives rise to a novel method, completing the practitioner's toolbox for problems where ADAM fails.

Subfields: Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Architecture

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “…leave (cid:15) at the common default value of 10 − 8 for Adam and 10 − 10 for RMSProp (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al.,…”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “SIGN SGD and ADAM are empirically known to generalize worse than SGD Wilson et al. [2017], Balles and Hennig [2018].”
Symbolic Discovery of Optimization Algorithms (2023, background): “There are a large number of handcrafted optimizers, mostly adaptive ones, introduced in recent years [2, 3, 7, 27, 58, 105].”
signSGD with Majority Vote is Communication Efficient And Byzantine Fault Tolerant (2018, methodology): “What’s more, in deep learning folklore sign based methods are known to perform well, indeed inspiring the popular RMSPROP and ADAM optimisers (Balles & Hennig, 2018), giving hope for D1.”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, methodology): “We performed extensive comparisons with other optimizers, including SGD [3], AdaBound [12], 466 Yogi [14], Adam [8], MSVAG [15], RAdam [16], Fromage [17] and AdamW [18].”
Adaptive Gradient Methods at the Edge of Stability (2022, background): “[3] argued that Adam should be viewed as a variant of sign gradient descent.”
Variance reduction properties of the reparameterization trick (2018, background): “Balles and Hennig (2018) also consider only the marginal variances when studying the eﬀect of the variability of the stochastic gradient on the Adam optimizer (Kingma and Ba, 2015).”
Limitations of the Empirical Fisher Approximation (2019, background): “…of decreasing in function value, while Schaul et al. [2013] proposed a diagonal rescaling based on the signal-to-noise ratio of each coordinate, Balles and Hennig [2018] identiﬁed these factors as optimal in that they minimize the expected error E Dg ( θ ) − ∇ L ( θ ) 22 for a diagonal matrix…”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “Recall that GD’s update in this setting is: A common simpliﬁcation of Adam that is more amenable to analysis (Balles & Hennig, 2018; Bernstein et al., 2018; Zhuang et al., 2020; Kunstner et al., 2023) is SignGD, which dates back to RProp (Braun & Riedmiller, 1992) that motivated RMSProp (Hinton et…”
Linear Convergence of Adaptive Stochastic Gradient Descent (2019, background): “κ is the condition number (2)Note that our results are for the norm version of AdaGrad (AdaGrad-Norm), which differs from the convergence of the diagonal version of AdaGrad and its variants (with momentum) (Balles and Hennig, 2018; Bernstein et al., 2018; Mukkamala and Hein, 2017; Chen et al., 2018)”
BackPACK: Packing more into backprop (2019, background): “…the batch has been found to be theoretically and practically useful for adapting hyperparameters like learning rates (Mahsereci & Hennig, 2017) and batch sizes (Balles et al., 2017), or regularize ﬁrst-order optimization (Le Roux et al., 2007; Balles & Hennig, 2018; Katharopoulos & Fleuret, 2018).”
A modified Adam algorithm for deep neural network optimization (2023, methodology): “We use HN_Adam, AdaBelief [30], Adam [8], SGD [33], Yogi [38], RAdam [40] and MSVAG [39] as learning algorithms during the training process of the ResNet18 deep network model.”