YellowFin and the Art of Momentum Tuning

By Jian Zhang, Ioannis Mitliagkas, Christopher R'e. 2017

113 citations recorded by the source corpus.

Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better test metrics. Motivated by this trend, we ask: can simple adaptive methods based on SGD perform as well or better? We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam. We then analyze its robustness to learning rate misspecification and objective curvature variation. Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD. YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly. We empirically show that YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to 3.28x in synchronous and up to 2.69x in asynchronous settings.

Subfields: Artificial Intelligence, Computer Science Applications, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “Others do not even report the value of used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019).”
Negative Momentum for Improved Game Dynamics (2018, methodology): “Some recent studies have emphasized the importance of momentum tuning in deep learning such as Sutskever et al. (2013), Kingma and Ba (2015), and Zhang and Mitliagkas (2017), however, none of them consider us-ing negative momentum.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, background): “Synchronization delays can be alleviated via asynchronous gradient aggregation (Recht et al., 2011; Cui et al., 2014; Gupta et al., 2016; Zhang et al., 2017; Dutta et al., 2018).”
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD (2018, neutral): “The idea of variable learning rate is related to the idea of momentum tuning in [42,54] and may have a similar effect of stabilizing the convergence of asynchronous SGD.”
The Importance of Being Recurrent for Modeling Hierarchical Structure (2018, methodology): “The FAN is trained with Adam for the language model objective and the YellowFin optimizer (Zhang et al., 2017)1 for the number prediction objective.”
Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods (2017, methodology): “which is being used extensively with mSGD for training deep neural networks [84,88,94].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ic[121,155,248,258], Natural Gradient[11,195] Optimization Algorithms First-Order[43,123,126,141,145,163,227] Second-Order[11,28,56,94,133,165,172] Evolutionary[169,192,234,242] Hyper-Parameter Search[13,89,91,118,131,163,169,219,256] Architecture Search: Reinforcement[12,193,264,268,269], Evolution”
On the adequacy of untuned warmup for adaptive optimization (2019, methodology): “This direction comes dangerously close to seeking the “holy grail” of an automatic learning rate tuner; existing attempts to devise such a method have achieved limited adoption as of yet (Li et al., 2017; Zhang et al., 2017; Baydin et al., 2018).”
An Improved Analysis of Stochastic Gradient Descent with Momentum (2020, methodology): “The initial learning rate of YellowFin is set to 0.1,1 and other parameters are set as their default values.”
Rethinking the Hyperparameters for Fine-tuning (2020, unknown): “Zhang & Mitliagkas (2017) proposed an automatic tuner for momentum and learning rate in SGD.”
Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning (2018, result): “ss. We can also apply the same idea to obtain Mini-batch Two-way Backtracking MMT (MBT-MMT) and Mini-batch Two-way Backtracking NAG (MBT-NAG). The above idea is compatible with recent research ([46], [50]), which suggests that traditional GD as well as MMT and NAG with a good learning rate can be on”
Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip (2019, methodology): “9 which is being used extensively with mSGD for training deep neural networks [211, 196, 185].”