Quasi-hyperbolic momentum and Adam for deep learning

By Jerry Ma, Denis Yarats. 2018

158 citations recorded by the source corpus.

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “…(cid:15) at the common default value of 10 − 8 for Adam and 10 − 10 for RMSProp (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al., 2019).”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “In the setting of smooth non-convex functions, Gitman et al. (2019) establish stability and asymptotic convergence results for the quasi-hyperbolic momentum method (Ma & Yarats, 2019), which can be viewed as interpolating between SGD and a stochastic momentum method.”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, methodology): “All systems feature ensembles of four to five transformer models that were trained using the quasi-hyperbolic Adam optimiser (Ma and Yarats, 2018).”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, background): “Other directions include Lookahead [42] which updates “fast” and “slow” weights separately, and is a wrapper that can combine with other optimizers; variance reduction methods [43, 44, 45] which reduce the variance in gradient; and LARS [46] which uses a layer-wise learning rate scaling.”
Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization (2019, methodology): “Moreover, originally motivated by variance reduction, QHAdam (Ma and Yarats, 2019) replaces both Adam’s moment estimates by quasi-hyperbolic terms and recovers Adam , Rmsprop and Nadam as particular cases (modulo the bias correction).”
Understanding the Role of Momentum in Stochastic Gradient Methods (2019, methodology): “In this paper, we focus on a class of algorithms captured by the general form of QHM [18]: d = (1 − βk)g + βkd, xk+1 = x − αk [ (1 − νk)g + νkd ] , (6)”
Learning Binary Trees via Sparse Relaxation (2020, methodology): “(13) and all neural network methods (DNDT, NDF, and NODE) using the Quasi-Hyperbolic Adam (Ma & Yarats, 2018) stochastic gradient descent method, with default parameters and batch size equal to 512 .”
An Improved Analysis of Stochastic Gradient Descent with Momentum (2020, methodology): “Other forms of momentum for stochastic optimization include PID Control-based methods [2], Accelerated SGD [12], and Quasi-Hyperbolic Momentum [17].”
Symbolic Discovery of Optimization Algorithms (2023, background): “Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].”
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (2019, methodology): “As an optimization method, we use the recent Quasi-Hyperbolic Adam with parameters recommended in the original paper (Ma & Yarats, 2018).”
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings (2020, methodology): “Accordingly, Nesterov’s Accelerated Gradient method is considered optimal in the sense that the convergence rate in (5) depends on √ Q rather than Q. The proof of (5) presented in Nesterov (2004) uses the method of estimate sequences.”