Improving Generalization Performance by Switching from Adam to SGD

By Nitish Shirish Keskar, Richard Socher. 2017

593 citations recorded by the source corpus.

Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Others do not even report the value of (cid:15) used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019).”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “Although the adaptive gradient methods such as Adam [40] tend to perform well in the initial phase of training, they are outperformed by SGD at later stages of training due to generalization issues [38, 10].”
Adaptive Gradient Methods with Dynamic Bound of Learning Rate (2019, methodology): “Reddi et al. (2018) proposed AMSG RAD to solve this problem but recent work has pointed out AMSG RAD does not show evident improvement over A DAM (Keskar & Socher, 2017; Chen et al., 2018).”
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (2018, background): “[Keskar and Socher, 2017] proposed to improve the generalization performance by switching from Adam to SGD.”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “However, these methods have been observed to hurt generalization performance [21, 10], and modern architectures often achieve the best results via SGD and hand-tuned learning rates [17, 42].”
CE-Net: Context Encoder Network for 2D Medical Image Segmentation (2019, methodology): “We use SGD optimization since recent studies [62], [63] show that SGD”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, background): “Further, works like [43] and [18] have shown cases where SGD (no momentum) and HB (classical momentum) generalize much better than RMSProp and ADAM with stochastic gradients.”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, methodology): “The generalization ability of DNN is also greatly affected by optimization methods [18, 45, 25].”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, methodology): “It is also well-known that 2nd-order methods, including K-FAC and diagonal methods like Adam/RMSprop, dont self-regularize nearly as much (e.g. Wilson et al., 2017; Keskar & Socher, 2017).”
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (2018, methodology): “…methods are often observed to converge faster, and hence may be used on problems which are more difficult to optimize (Graves, 2013), but can give worse generalisation performance than non-adaptive methods (Keskar & Socher, 2017; Loshchilov & Hutter, 2017; Wilson et al., 2017; Luo et al., 2019).”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, methodology): “Recent progress tries to combine the benefits of 23 both, such as switching from Adam to SGD either with a hard schedule as in SWATS [11], or with a 24 smooth transition as in AdaBound [12].”
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks (2019, methodology): “One solution to this problem, proposed by Keskar and Socher [9], is to use Adam during the initial stage and switch to SGD in the later stage of training.”
NAMSG: An Efficient Method For Training Neural Networks (2019, methodology): “Methods to address this issue include, e.g., data argumentation, increasing the regularization parameter or step sizes, and switching to SGD [Keskar and Socher, 2017, Luo et al., 2019].”
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning (2020, background): “So m ( W SGD-M ) is much larger than m ( W Adam ) .”
Deep learning for cellular image analysis (2019, methodology): “Recent work suggests that networks trained with stochastic gradient descent with momentum have better performance with respect to generalization 175,176 .”
Projection-Free Adaptive Gradients for Large-Scale Optimization (2020, background): “…and Hinton, 2012), AdaDelta (Zeiler, 2012), Adam (Kingma and Ba, 2015), AMSGrad (Reddi et al., 2018), and, e.g., AdaBound (Luo et al., 2019; Keskar and Socher, 2017), with each new variant addressing some flaws in the previous ones: vanishing step-sizes, incomplete theory, generalization…”
Information-Theoretic Local Minima Characterization and Regularization (2019, methodology): “In speciﬁc, very similar to our setup, Keskar & Socher (2017) demonstrates that SGD+Momentum consistently outperforms the others on ResNet and DenseNet for CIFAR-10 and CIFAR-100.”
Adaptive Gradient Methods at the Edge of Stability (2022, methodology): “Implicit bias of adaptive gradient methods Absent well-tuned regularization, adaptive gradient methods have been reported to generalize worse than non-adaptive optimizers [42, 22, 28].”