Adaptive Gradient Methods with Dynamic Bound of Learning Rate

By Liangchen Luo, Yuanhao Xiong, Yan Liu, Xu Sun. 2019

693 citations recorded by the source corpus.

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at this https URL .

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Others do not even report the value of (cid:15) used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019).”
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “[32] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “We compare our methods against Adam [39], which is the most common adaptive method, and other methods that report better performance than Adam: coin-betting [60], L41 [68], and Adabound [51].”
On the Variance of the Adaptive Learning Rate and Beyond (2019, methodology): “…al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen et al., 2018), or adding range constraints (Luo et al., 2019)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba et al., 2016; Ioffe & Szegedy, 2015).”
Adaptive Federated Optimization (2020, neutral): “, 2019), AdaBound (Luo et al., 2019)).”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “Luo et al. [2019] show that by bounding the adaptive step-sizes in ADAM leads to closing the generalization gap.”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, background): “, 2018) and (6) AdaBound (Luo et al., 2019).”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, methodology): “Adaptive gradient methods are a popular method for deep learning [14, 43, 37, 23, 29] that adaptively choose different step sizes for different parameters.”
On the adequacy of untuned warmup for adaptive optimization (2019, background): “Partially adaptable meth-ods (Chen & Gu, 2018; Keskar & Socher, 2017; Luo et al., 2019), which interpolate between the full adaptivity of Adam and the non-adaptivity of SGD, may hold more promise for improving late-stage convergence.”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, methodology): “We performed extensive comparisons with other optimizers, including SGD [3], AdaBound [12], 466 Yogi [14], Adam [8], MSVAG [15], RAdam [16], Fromage [17] and AdamW [18].”
Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data (2020, background): “…for the second-order momentum vt and the related adaptive learning rate in Adagrad (Duchi et al. (2011)), Adadelta (Zeiler (2012)), Rmsprop (Tieleman and Hinton (2012)), Adam (Kingma and Ba (2014)), AMSGrad (Reddi et al. (2018)), Yogi (Zaheer et al. (2018)) and AdaBound (Luo et al. (2019)).”
Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints (2019, methodology): “Adaptive learning rate methods (Tieleman & Hinton, 2012; Kingma & Ba, 2015; Reddi et al., 2018; Luo et al., 2019) adjust the learning rate according to the local statistics of the cost surface.”
Large-Scale Generative Data-Free Distillation (2020, background): “ned network, while leaving the activations of internal layers unconstrained. Previous studies have observed that different layers of a deep convolutional network are likely to perform different tasks [18,34,38], i.e., lower layers tend to detect low-level features such as edges and curves, while hig”