Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation

By Hamid Reza Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, Richard S. Sutton. 2009

307 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “issues with Q-learning have been partially addressed by gradient temporal-difference methods. These methods are proven to converge when evaluating a ﬁxed policy with a nonlinear function approximator [14]; or when learning a control policy with linear function approximation using a restricted varian”
High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, neutral): “Some candidates for such an error metric might include the Bellman error or projected Bellman error, as described in [3].”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “Unfortunately, the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence (Bhatnagar et al., 2009).”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Convergent backups [47, 46, 32] were developed.”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “Thus, the existing TD learning algorithms with convergence guarantees under off-policy data (Maei et al. 2009), (Sutton et al.”
Linear Off-Policy Actor-Critic (2012, methodology): “The standard way of avoiding the limitations of actionvalue methods is to use policy-gradient algorithms (Sutton et al., 2000) such as actor-critic methods (e.g., Bhatnagar et al., 2009).”
Soft Actor-Critic Algorithms and Applications (2018, background): “Unfortunately, the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence (Bhatnagar et al., 2009).”
GQ(lambda): A general gradient algorithm for temporal-difference prediction learning with eligibility traces (2010, background): “Recently, Sutton et al. (2009b) and Maei et al. (2009) introduced a new family of gradient TD methods in which function approximation is much more straightforward than conventional methods.”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “Compared to nonlinear GTD2/TDC (Maei et al., 2009), our method is simpler (without having to do a local linear expansion) and empirically more effective (as demonstrated in the next section).”
Policy Gradient Methods for Off-policy Control (2015, unknown): “Maei [6] present the ﬁrst gradient based TD algorithm that converges in this case.”
Distributed Policy Evaluation Under Multiple Behavior Strategies (2013, unknown): “Now, the analysis of the convergence of reinforcement learning algorithms is usually challenging even for the single-agent case, and studies are often restricted to the case of diminishing step-sizes [9], [10], [23], [24] since this condition allows to use stochastic approximation results [25], [26]”
Weighted Bellman Equations and their Applications in Approximate Dynamic Programming ∗ (2012, background): “The ideas are applicable in general-space MDP with discounted or undiscounted cost criteria, and with nonlinear function approximation [MSB09].”
Algorithms for Reinforcement Learning (2010, methodology): “More recently, Bhatnagar et al. (2009) proposed several two-timescale algorithms and proved the convergence of the policy parameters to a neighborhood of the local maxima of the objective function when the critic uses TD(0)-like updates.”
Reinforcement Learning under Model Mismatch (2017, background): “The mean squared projected Bellman equation ( MSPBE ) loss function considered by [7] can then be deﬁned as where we now project to the the tangent space T M θ .”