Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

By Mahesh Chandra Mukkamala, Matthias Hein. 2017

279 citations recorded by the source corpus.

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show $\sqrt{T}$-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.

Subfields: Artificial Intelligence, Computational Mechanics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Management Science and Operations Research

Citations in the corpus

Generalized Byzantine-tolerant SGD (2018, background): “Its variants are widely used in training deep neural networks (Kingma & Ba, 2014; Mukkamala & Hein, 2017).”
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization (2018, neutral): “, 2016), AdaBatch (Defossez and Bach, 2017), SC-Adagrad (Mukkamala and Hein, 2017), AMSGRAD (Reddi et al.”
A Sufficient Condition for Convergences of Adam and RMSProp (2018, background): “Prior to that, the convergence rate of RMSProp [19] has already been established in the convex stochastic setting by employing similar parameters to those of AdamNC [20].”
On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks (2018, methodology): “By replacing the constant parameter β by β t = 1 − with γ ∈ (0 , 1], RMSProp has been shown to be convergent in the convex stochastic setting [28].”
Escaping Saddle Points with Adaptive Gradient Methods (2019, result): “Mukkamala and Hein [2017] prove improved convergence bounds for Adagrad in the online strongly convex case; they prove similar results for RMSProp, but only in a regime where it is essentially the same as Adagrad.”
Phocas: dimensional Byzantine-resilient stochastic gradient descent (2018, background): “Its variants are widely used in training deep neural networks [9, 17].”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, background): “Further, [25] shows convergence in the online convex setups of variants of RMSProp that use timedependent hyperparameters.”