Accelerated Gradient Methods for Stochastic Optimization and Online Learning

By Chonghai Hu, James T. Kwok, Weike Pan. 2009

186 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Proximal Stochastic Gradient Method with Progressive Variance Reduction (2014, background): “It is a slight generalization of [HKP09, Lemma 3], and we give the proof in Appendix A for completeness.”
Optimization with Sparsity-Inducing Penalties (2011, background): “1) may also be tackled with stochastic optimization approaches, which has been the focus of much recent research [21, 23, 44, 61, 126, 153].”
Katyusha: the first direct acceleration of stochastic gradient methods (2016, background): “Some authors focus on obtaining a mixed running time for instance of the form O ( L/ √ ε + σ/ε 2 ) in the presence of stochastic gradient with variance σ [16, 18].”
Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization (2013, neutral): “See for example [10, 9, 4] and the references therein.”
StopWasting My Gradients: Practical SVRG (2015, methodology): “We assume that ‖xt−x‖ ≤ Z for all t, which has been used in related work [10] and is reasonable because of the coercity implied by strong-convexity.”
A variational perspective on accelerated methods in optimization (2016, background): “Introduced by Nesterov in 1983 [24] in the context of gradient descent, the acceleration idea has been extended to a wide range of other settings, including composite optimization [27, 33, 7], stochastic optimization [12, 18], nonconvex optimization [11, 21], and conic programming [19].”
Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization (2013, methodology): “Some previous works are closely related to this scheme: the online EM algorithm for latent data models [5,21] and the online matrix factorization technique of [19] involve for instance surrogate functions updated in a similar fashion.”
On stochastic proximal gradient algorithms (2014, background): “Other authors have proposed algorithms based on proximal operators to better exploit the smoothness of ` and the properties of g (Hu et al. (2009); Xiao (2010)).”
On the influence of momentum acceleration on online learning (2016, methodology): “…Proakis, 1974; Sharma et al., 1998; Shynk and Roy, June 1988; Roy and Shynk, 1990; Tugay and Tanik, 1989; Bellanger, 2001; Wiegerinck et al., 1994; Hu et al., 2009; Lan, 2012; Ghadimi and Lan, 2012; Zhong and Kwok, 2014) and applied, for example, to problems involving the tracking of chirped…”
Stochastic Heavy ball (2016, methodology): “Comparisons with other algorithms Figure 3 compares the HBF-algorithm with several stochastic optimization algorithms: the standard Robbins-Monro stochastic gradient descent (SGD) introduced in [RM51] and several second order algorithms: the “optimal” RuppertPolyak averaging algorithm (see [PJ92, Ru”
Minimizing finite sums with the stochastic average gradient (2013, methodology): “radient magnitudes. Alternately, if we split the convergence rate into a deterministic and stochastic part, these methods can improve the dependency of the convergence rate of the deterministic part [Hu et al., 2009, Ghadimi and Lan, 2010, Xiao, 2010]. However, we are not aware of any existing metho”
Stochastic Alternating Direction Method of Multipliers (2013, methodology): “Stochastic and online variants of these methods have also been developed recently, mainly in the large-scale machine learning and optimization literature (Langford et al., 2009; Lan, 2010; Lan & Ghadimi, 2011; Duchi & Singer, 2009; Hu et al., 2009; Xiao, 2010).”
A Stochastic forward-backward splitting method for solving monotone inclusions in Hilbert spaces (2014, background): “lues which is optimal both with respect to the smooth component and the non-smooth term. Similar accelerated proximal gradient algorithms have been also studied in the machine learning community, see [32, 55], and also [10, 50, 51, 58]. Problem 4.8. Let β ∈]0,+∞[, let G ∈ Γ 0(H), and let L: H → R be”
Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization (2009, background): “(36) Such a result was obtained by Hu et al. (2009). Their algorithm can be viewed as an accelerated version of the Fobos method (28).”
Online Adaptive Methods, Universality and Acceleration (2018, methodology): “While each of these issues was separately resolved in [17, 16, 33], and respectively in [25]; it was unknown whether there exists an accelerated method that addresses both issues.”
Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity (2014, background): “Toaccelerate the convergence, some variance-reduced stochastic gradient algorithms, e.g., proximal stochastic variance-reduced gradient (Prox-SVRG) algorithm, have recently been proposed to solve strongly convex problems.”