Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

By Jinghui Chen, Yuan Cao, Quanquan Gu. 2018

222 citations recorded by the source corpus.

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On the Variance of the Adaptive Learning Rate and Beyond (2019, neutral): “…such as gradient clipping (Bengio et al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen et al., 2018), or adding range constraints (Luo et al., 2019)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba…”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “Chen and Gu [2019] interpolate between SGD and ADAM using a new hyper-parameter p and show that tuning this can recover performance of SGD. Zaheer et al. [2018] introduce a new adaptive algorithm which is closer to Adagrad Duchi et al. [2011].”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “Although the adaptive gradient methods such as Adam [38] tend to perform well in the initial phase of training, they are outperformed by SGD at later stages of training due to generalization issues [36, 9].”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “However, these methods have been observed to hurt generalization performance [21, 10], and modern architectures often achieve the best results via SGD and hand-tuned learning rates [17, 42].”
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout (2020, methodology): “Apart from the traditional optimization methods [8, 19, 32, 33, 49], there has been a research thrust on developing different ways to apply gradients to deep networks [2, 7, 10, 15, 36, 45, 50].”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, unknown): “Restrictions apply.”
TAG: Task-based Accumulated Gradients for Lifelong learning (2021, background): “These optimizers tend to ‘over-adapt’ on the most recent batch and hence suffer from poor generalization performance [27, 11, 40].”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, background): “AdaBound [12] achieves this goal on Cifar, yet its performance on 520 ImageNet is still inferior to SGD [22].”
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2019, background): “For example, ever since Zhang et al. [65] demonstrated the effective capacity of several successful neural network architectures is large enough to memorize random labels, the community sees a prosperity of many discussions about this apparent ”paradox” [61, 15, 17, 15, 11].”
Extreme Tensoring for Low-Memory Preconditioning (2019, methodology): “Some recent preconditioning methods are not based on second moments [BZVL17, CG18, BWAA18], and fall beyond our scope.”
Adaptive Gradient Methods at the Edge of Stability (2022, methodology): “…training; (b) AdamW [29], which employs decoupled weight decay; (c) Adafactor [38], which maintains a factored layer-wise preconditioner; (d) Amsgrad [37], which ensures that the preconditioner is entrywise non-decreasing over time; (e) Padam [5], which employs an exponent smaller than (we use 0 .”
Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization (2019, background): “We also mention that most of the theoretical bounds depend on the dimension of the parameter (Reddi et al., 2018; Zhou et al., 2018; Chen et al., 2018; Zou et al., 2019; Chen et al., 2019; Luo et al., 2019).”
A modified Adam algorithm for deep neural network optimization (2023, result): “The results of the compared algorithms are taken the same as in [30, 66].”
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (2019, background): “By evaluating on a wide range of 9 diverse tasks, we contribute to the debate of adaptive vs. non-adaptive optimizers (Wilson et al., 2017; Shah et al., 2018; Chen & Gu, 2018; Choi et al., 2019).”
Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network (2019, background): “Adaptive Gradient Methods Adaptive Gradient (AdaGrad) Methods, first introduced independently by Duchi et al. [2011] and McMahan and Streeter [2010], are now widely used in practice for online learning due in part to their robustness to the choice of stepsize. The first convergence guarantees, prove”