On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks

By Fangyu Zou, Li Shen. 2018

16 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “…(cid:15) at the common default value of 10 − 8 for Adam and 10 − 10 for RMSProp (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al., 2019).”
A Sufficient Condition for Convergences of Adam and RMSProp (2018, methodology): “[28] recently proposed a class of new surrogates called AdaUSM to approximate Adam and RMSProp by integrating weighted AdaGrad with uniﬁed heavy ball and Nesterov accelerated gradient momentums.”
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (2018, background): “Other works analyzing the convergence of adaptive methods are (Ward et al., 2018; Li & Orabona, 2018; Zou & Shen, 2018; Zhou et al., 2018).”
Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (2019, background): “One elegant approach is variance reduction (e.g. Schmidt et al., 2017; Johnson and Zhang, 2013; Defazio et al., 2014; Bach and Moulines, 2013; Koneˇcn`y and Richt´arik, 2013; Xiao and Zhang, 2014; Gong and Ye, 2014; Fang et al., 2018; Zhou et al., 2018b).”
Why ADAM Beats SGD for Attention Models (2019, background): “Since then, many works studied different interesting aspects of adaptive methods, see (Ward et al., 2018; Li & Orabona, 2018; Zhou et al., 2018a; Staib et al., 2019; Chen et al., 2018; Zou & Shen, 2018; Zhou et al., 2018b; Agarwal et al., 2018; Zou et al., 2019).”