Stochastic Approximation: A Dynamical Systems Viewpoint

By Vivek Shripad Borkar. 2008

1,413 citations recorded by the source corpus.

Subfields: Finance, Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Finance

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Generalized Off-Policy Actor-Critic (2019, background): “Theoretically, a convergent analysis of Geoff-PAC involving compatible function assumption (Sutton et al., 2000) or multi-timescale stochastic approximation (Borkar, 2009) is also worth further investigation.”
Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning (2016, background): “For textbook references on stochastic approximation, please see (Kushner and Yin, 2003; Benveniste et al., 2012; Borkar, 2008; Bertsekas and Tsitsiklis, 1989).”
Linear Off-Policy Actor-Critic (2012, methodology): “Following previous oﬀ-policy gradient proofs (Maei, 2011), we study the behavior of the ordinary diﬀerential equation The two updates (for the actor and for the critic) are not independent on each time step; we analyze two separate ODEs using a two timescale analysis (Borkar, 2008).”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, background): “A key property of this class of GTD algorithms is that they are asymptotically off-policy convergent, which was shown using stochastic approximation [Borkar, 2008].”
An online primal-dual method for discounted Markov decision processes (2016, methodology): “[3], by Borkar [7], by Bertsekas and Tsitsiklis [6]).”
Q-learning and enhanced policy iteration in discounted dynamic programming (2010, background): “, O41/k5): For all ` ∈ Rk, let `1 k have a common value k, and select all state-control pairs 4i1 u5 “comparably often” in the sense that the fraction of times 4i1 u5 is selected for iteration is nonzero in the limit (see Borkar [17]).”
Algorithms for Reinforcement Learning (2010, background): “This makes TDC a member of the family of the so-called two-timescale stochastic approximation algorithms (Borkar, 1997, 2008).”
Distributed Policy Evaluation Under Multiple Behavior Strategies (2013, background): “llenging even for the single-agent case, and studies are often restricted to the case of diminishing step-sizes [8], [9], [22], [23] since this condition allowsto use stochastic approximation results [24] to guarantee convergence. For a distributed algorithm, the analysis becomes more demanding beca”
Randomized Linear Programming Solves the Discounted Markov Decision Problem In Nearly-Linear Running Time (2017, methodology): “Stochastic first-order methods originate from the stochastic approximation method for the root finding problem; see [18] [2] [6].”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, methodology): “We employ here the ODE based analysis as proposed in (Borkar, 2008; Kushner and Clark, 2012).”
Central Limit Theorems for Stochastic Approximation with controlled Markov chain dynamics (2013, background): “Note that this paper is devoted to convergence rates so that sufficient conditions for the convergence is out of the scope of the paper; for convergence, the interested reader can refer to [4, 11, 3, 8, 2, 6].”
Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication (2008, background): “This should not be confused with (centralized) stochastic algorithms with coupling (see [37]), where a quickly switching parameter influences the relatively slower dynamics of another state, leading to averaged dynamics.”
Empirical Dynamic Programming (2013, background): “Furthermore, while many techniques for establishing convergence have been developed [11], including the o.”
Linear Stochastic Approximation: How Far Does Constant Step-Size and Iterate Averaging Go? (2018, background): “, independence, or mixing, in addition to bounded moments, if the stepsize sequence converges to zero at an appropriate rate, convergence of {θt}t≥0 to θ∗ can be guaranteed in various senses [2, 3].”
Learning the Variance of the Reward-To-Go (2016, methodology): “Proof The proof is based on representing the algorithm as a stochastic approximation, and uses a result of Borkar (2008) to show that the iterates asymptotically track a certain ordinary differential equation (ODE).”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, positive): “However, only a few algorithms consider safety constraints, and safe MARL still requires a lot more attention.”