Accelerated Stochastic Approximation

By Bernard Delyon, Anatoli B. Juditsky. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Minimizing finite sums with the stochastic average gradient (2013, methodology): “In particular, accelerated SG methods use a constant step size by default, and only decrease the step size on iterations where the inner-product between successive gradient estimates is negative [14,25].”
Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta (1992, methodology): “The new algorithm is an extension of the Delta-Bar-Delta algorithm (Jacobs 1988; Sutton 1982; Barto & Sutton 1981; Kesten 1958) such that it is applicable to incremental tasks—supervised learning tasks in which the examples are processed one by one and then discarded.”
Handbook of Simulation Optimization (2014, methodology): “For the time being, we recommend estimating the true mean response for a given input combination such that a relative precision of (say) 10 % has a 90 % probability, using the method detailed in [31].”
On stochastic gradient and subgradient methods with adaptive steplength sequences (2011, methodology): “sted a technique where the steplength sequence adapts to the observed data, which was further extended by Kushner and Gavin [51] to the multidimensional regime,while its accelerations were studied in [52]. Sacks [49] proved that, under suitable conditions, a choice of the form a=k(where kis the iter”
A tutorial on adaptive MCMC (2008, methodology): “The technique was first described in Kesten (1958) and relies on the remark that, for example, an alternating sign for {α̂θi − ᾱ∗} in (22) is an indication that {θi} is oscillating around (a) solution(s), whereas a constant sign suggests that {θi} is, roughly speaking, still far from the…”
A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets (2012, methodology): “In particular, accelerated SG methods use a constant step size by default, and only decrease the step size on iterations where the inner-product between successive gradient estimates is negative Kesten [1958], Delyon and Juditsky [1993].”
An Overview of Stochastic Approximation (2015, background): “An extension of Kesten’s rule to higher dimensions is discussed in [11].”
Reinforcement Learning in Situated Agents : Some Theoretical Problems and (1999, background): “According to Jacobs (1988), Kesten (1958) rstsuggested that an alternating error sign could beused as an indication that a step-size parametershould be reduced in the context of a stochasticapproximation method, and that Saridis (1970) ex-tended the idea to both increasing and decreasingthe a…”
Barzilai-Borwein Step Size for Stochastic Gradient Descent (2016, background): “This kind of idea can be traced back to [11].”