Stochastic approximation with two time scales

By Vivek Shripad Borkar. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Convergence of RUDDER learning can under standard assumptions be proven by the stochastic approximation for two time-scale update rules [12, 18].”
Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring (2012, neutral): “The theory of adaptive MCMC (Andrieu & Thoms, 2009) or two time scale stochastic approximations (Borkar, 1997) might hold the key to such a proof which we leave for future work.”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, methodology): “The Appendix A provides a complementary analysis form the perspective of stochastic approximation (Heusel et al., 2017; Borkar, 1997).”
Actor-Critic Algorithms (1999, result): “results of [12, 6 , 14], this need not pose any problems, as long as the actor parameters are updated on a slower time-scale.”
Reinforcement Learning: A Tutorial Survey and Recent Advances (2009, unknown): “Step 4a is as follows: Tr ← Tr + r(i, a, j); Tt ← Tt + t(i, a, j); and ρ ← (1 − β)ρ + βTr/Tt, where β, the step size used for updating ρ, should be smaller than µ (Borkar, 1997).”
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (2017, methodology): “The next theorem has been proved in the seminal paper of Borkar 1997 [7].”
Stochastic Heavy ball (2016, unknown): “For optimization procedures, this two-scales setting appears in [Bor97] (see also [Bor08]) where under an appropriate control of the noise and some uniqueness conditions, the a.”
An analysis of reinforcement learning with function approximation (2008, background): “The analysis in (Borkar, 1997) could then be replicated.”
Online Model Learning Algorithms for Actor-Critic Control (2015, background): “For those algorithms, convergence is ensured as long as the actor is updated on a slower timescale than the critic (Borkar, 1997; Konda and Tsitsiklis, 2003).”
Geometrical Insights for Implicit Generative Modeling (2017, background): “Such algorithms belong to the general class of stochastic algorithms with two time scales [9, 29].”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, methodology): “Further, it is analytically admissible (Borkar, 1997) to consider the slow timescale stochastic recursion (i.e., the neural network SGD) to be quasi-stationary (i.e., θ̄t ≡ θ̄, ∀t ∈ N), while analyzing the asymptotic behaviour of the relatively faster timescale stochastic recursion (17).”
Algorithms for Reinforcement Learning (2010, background): “This makes TDC a member of the family of the so-called two-timescale stochastic approximation algorithms (Borkar, 1997, 2008).”