Rates of Convergence of Adaptive Step-Size of Stochastic Approximation Algorithms

By Sally Shao, Percy P. C. Yip. 2000

4 citations recorded by the source corpus.

We propose a new adaptive algorithm with decreasing step-size for stochastic approximations. The use of adaptive algorithms in various applications is widely spread across fields such as system identification and adaptive control. We analyze the rate of convergence of the proposed algorithms. An averaging algorithm, on its optimality of the rate of convergence, is used to control the step sizes. Our proofs are based on recent results in stochastic approximations and Gaussian Approximation Theorem.

Subfields: Artificial Intelligence, Computational Mechanics, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Online Learning Rate Adaptation with Hypergradient Descent (2017, background): “Similarly, Shao and Yip [24] propose a way of controlling the learning rate of a main algorithm by using an averaging algorithm based on the mean of a sequence of adapted learning rates, also investigating rates of convergence.”