On Empirical Comparisons of Optimizers for Deep Learning

By Dami Choi, Christopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, George E. Dahl. 2019

304 citations recorded by the source corpus.

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, neutral): “Evaluations of optimization methods should account for all experimentation efforts required to tune optimizer hyperparameters, not just the method performance after tuning [57], [58].”
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (2019, methodology): “In a concurrent study, Choi et al. (2019) show that there exists a hierarchy among optimizers such that some can be viewed as speciﬁc cases of others and thus, the general optimizer should never under-perform the special case (with appropriate hyperparameter settings).”
A Generalizable Approach to Learning Optimizers (2021, background): “While past work has noted that a more general optimizer should be able to outperform one they can approximate [10], the bounds on our inner optimizer’s hyperparameter values prevent it from doing so (Appendix I).”