Gradient temporal-difference learning algorithms

By Richard S. Sutton, Hamid Reza Maei. 2011

176 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “We ﬁrst rewrite the update for d t as The rest is the same as the proof of Theorem 2 in page 44 of Maei (2011) without changing notations except for that we need to verify that the real parts of all eigenvalues of G are negative.”
Deep Reinforcement Learning and the Deadly Triad (2018, background): “ “the deadly triad” due to this possibility of divergence (Sutton and Barto, 2018). Partial solutions have been proposed (Baird, 1995; Precup and Sutton, 2001; Sutton et al., 2008; Maei et al., 2009; Maei, 2011; Sutton et al., 2016), but they have mostly not been extended to non-linear arXiv:1812.02”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, methodology): “o optimal, and even a= 0:25 produced a much more reasonable solution than a= 0. We ran a similar experiment, this time using value estimates from a critic trained with Gradient TD, called GTD() [Maei, 2011] to examine whether the impact of avalues with the actual (nonidealized) ACE algorithm pers”
Smoothed Dual Embedding Control (2017, background): “Maei, H. R. (2011). Gradient Temporal-Difference Learning Algorithms. PhD thesis, University of Alberta, Edmonton, Alberta, Canada. Mahadevan, S., Liu, B., Thomas, P. S., Dabney, W., Giguere, S., Jacek, N., Gemp, I., and Liu, J. (2014). Proximal reinforcement learning: A new theory of sequential dec”
On Convergence of Emphatic Temporal-Difference Learning (2015, background): “See e.g., (Maei, 2011; Bertsekas and Yu, 2009; Geist and Scherrer, 2014; Dann et al., 2014).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “For example, the updates of GTD(λ) and GQ(λ) (Maei 2011) with λ = 1 will tend to infinite variance as t→∞ on Baird’s counterexample and on the example in Figures 1 and 3(left).”
Linear Off-Policy Actor-Critic (2012, methodology): “Following previous offpolicy gradient proofs (Maei, 2011), we study the behavior of the ordinary differential equation u̇(t) = u(h(u(t),v))”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, methodology): “For learning the critic Q πM µ ( s, a ) , we can use any off-policy Temporal Difference (Bhatnagar et al., 2009; Maei, 2011) or Q-learning algorithm (Watkins and Dayan, 1992).”
Emphatic Temporal-Difference Learning (2015, methodology): “Compared to convergent gradient-based TD algorithms (Maei 2011), ETD(λ) is simpler and easier to use; it has only one learned parameter vector and one step-size parameter.”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “2 in Maei [2011]. Lemma 1.”
A Kernel Loss for Solving the Bellman Equation (2019, neutral): “lar algorithms like PCL (Nachum et al., 2017, 2018). Another line of work seeks alternative objective functions, the minimization of which leads to desired value functions (e.g., Sutton et al., 2009; Maei, 2011; Liu et al., 2015; Dai et al., 2017). Most existing works are either for linear approxima”
Weighted importance sampling for off-policy learning with linear function approximation (2014, background): “Most off-policy learning methods correct only the discrepancy between the target and the sample conditional distributions of outputs (Hachiya et al. 2009, Maei & Sutton 2010, Yu 2010, Maei 2011, Geist & Scherrer 2014, Dann et al. 2014).”
True Online Emphatic TD(λ): Quick Reference and Implementation Guide (2015, background): “We consider the setting of general value functions, or GVFs (Maei & Sutton 2010, Sutton et al. 2011, White 2015, Sutton, Mahmood & White 2015).”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, methodology): “1 It converges to mean-square TD error (MSTDE), as proven in [75].”
Natural Gradient Deep Q-learning (2018, background): “formance in a suite of challenging Atari games [Mnih et al., 2013]. Because Q-learning, as the control extension of TD algorithms [Sutton, 1988], is not truly a stochastic gradient descent algorithm [Maei, 2011], convergence of the algorithm with non-linear function approximators is poorly understoo”
Off-policy learning based on weighted importance sampling with linear computational complexity (2015, methodology): “Modern off-policy algorithms with O(n) computational complexity, such as GTD( ) (Maei 2011), GQ( ) (Maei & Sutton 2010), and true online GTD( ) (van Hasselt, Mahmood & Sutton 2014), all use the ordinary variant of importance sampling and can suffer severely due to the problem of large variance (Defa”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “The non-singularity of A(θ) and Ã(θ) for each fixed θ is commonly assumed (Sutton et al., 2009a;b; Maei, 2011).”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “The nonsingularity in Assumption 2 is commonly assumed in GTD methods (Sutton et al., 2009b;a; Maei, 2011) and can be satisﬁed by using linearly independent features.”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, methodology): “Therefore, the RL paradigm includes controlling and prediction; it also provides means to represent highly expressive knowledge using General Value Functions (GVFs) from sensorimotor streams [14].”