RMSProp and equilibrated adaptive learning rates for non-convex optimization

By Yann Dauphin, Harm de Vries, Yoshua Bengio. 2015

492 citations recorded by the source corpus.

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so-called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “…some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques attempt to approximate the inverse Hessian in a reduced way, e.g., by considering…”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “We use a learning rate of 0.001 trained with rmsprop (Dauphin et al., 2015).”
Analyzing and Improving the Image Quality of StyleGAN (2019, methodology): “To avoid explicit computation of the Jacobian matrix, we use the identity Jwy = ∇w(g(w) · y), which is efficiently computable using standard backpropagation [5].”
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “More recently, adaptive methods (Duchi et al., 2011; Liu et al., 2019; Kingma & Ba, 2015; Bengio, 2015; Vaswani et al., 2019b; Li & Orabona, 2019; Ward et al., 2019) that adjust the step-size on the fly have become wide-spread and are particularly beneficial when training deep neural networks.”
Recurrent convolutional neural networks for object-class segmentation of RGB-D video (2015, methodology): “We learn the parameters of our network using Root Mean Square Propagation (RMSProp), which is a variant of Resilient Propagation (RProp) suitable for mini-batch learning [15].”
No More Pesky Learning Rate Guessing Games (2015, methodology): “Dauphin et al.[21] argue that the difficulty in minimizing the loss arises from saddle points rather than poor local minima.”
Session-based Recommendations with Recurrent Neural Networks (2015, unknown): “We experimented with both rmsprop (Dauphin et al., 2015) and adagrad (Duchi et al., 2011).”
Loss-aware Binarization of Deep Networks (2016, background): “This is also called preconditioning in the literature (Dauphin et al., 2015a).”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “dialated convolution Batch normalization Used pyramid pooling module for aggregating multi-scale features FC-DensNet Initialized the model with HeUniform[120] and trained it with RMSprop dataset[121] Used dropout of 0.2 Used the model parameters eciently Gated -SCNN End to end trainable Appli”
Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors (2017, methodology): “The fully connected fusion layer consists of 64 units and was trained with an RMSprop optimiser [8] with learning rate of 10−4.”
MHCflurry: open-source class I MHC binding affinity prediction (2017, unknown): “Training proceeds using the RMSprop optimizer[19] until the accuracy on the test set has not improved for ten epochs.”
SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction (2018, methodology): “The parameters of our proposed network are trained with a RMSprop optimizer [5] and the learning rate is set to 0.”
SGDR: Stochastic Gradient Descent with Restarts (2016, background): “…some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-ofthe-art optimization techniques tend to approximate the inverse Hessian in a reduced way, e.g., by considering only…”
Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring (2017, unknown): “Optimization We use RMSprop (Dauphin et al., 2015) as our optimizer to train the whole model.”
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks (2015, background): “07666v1 [ s t a t .”
Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, methodology): “Our implementation of Hessian diagonal computation was inspired by Dauphin et al. (2015) work, where the technique proposed by Bekas et al. (2007) was used to evaluate SGD preconditioned with the Jacobi preconditioner.”