AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

By Juntang Zhuang, Tommy M. Tang, Yifan Ding, Sekhar Chandra Tatikonda, Nicha C. Dvornek, X. Papademetris, James S. Duncan. Neural Information Processing Systems, 2020

669 citations recorded by the source corpus.

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability.We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer. Code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1e04ca1998c04040c9c10685fc0daa4ecc13855b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multimodal Fusion with Co-Attention Networks for Fake News Detection (2021, methodology): “We use Adam (Kingma and Ba, 2014) and AdaBelief (Zhuang et al., 2020) as optimizers on Twitter and Weibo datasets, respectively, to seek the optimal parameters of our model.”
Domain-Independent Dominance of Adaptive Methods (2019, methodology): “CIFAR-10 Test Err% CIFAR-100 Test Err % ImageNet Val Err % Penn Treebank Test BPC ↓ Penn Treebank Test BPC ↓ CIFAR-10 FID ↓ Model WRN 28-10 WRN 28-10 ResNet-50 3xLSTM(300) 3xLSTM(1000) GGAN SGD 3.86 (0.08) 19.05 (0.24) 24.01 1.404 1.237 (0.000) 133.0 Adam 3.64 (0.06) 18.96 (0.21) 23.45 1.377 1.182 (”
A modified Adam algorithm for deep neural network optimization (2023, methodology): “The modiﬁed algorithm, HN_Adam, is tested by using it to train a deep convolutional neural network using two different datasets CIFAR-10 [30] and MNIST [13].”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, methodology): “For Adan, we utilize its default weight 1 The reported results in [15] slightly differ from the those in [38] because of their different settings for LSTM and training hyper-parameters. decay (0.02) and β s ( β 1 = 0 .”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, positive): “…that GD’s update in this setting is: A common simpliﬁcation of Adam that is more amenable to analysis (Balles & Hennig, 2018; Bernstein et al., 2018; Zhuang et al., 2020; Kunstner et al., 2023) is SignGD, which dates back to RProp (Braun & Riedmiller, 1992) that motivated RMSProp (Hinton et al.,…”
Symbolic Discovery of Optimization Algorithms (2023, methodology): “, 2020), NAdam (Dozat, 2016), AdaBelief (Zhuang et al., 2020) and AMSGrad (Reddi et al.”