A convergent O ( n ) algorithm for off-policy temporal-difference learning with linear function approximation

By Richard S. Sutton, Csaba Szepesvari, Hamid Reza Maei. 2008

199 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Residual gradient (RG) and gradient temporal difference (GTD) methods were introduced in (Baird, 1995; Sutton et al., 2009a,b).”
When to use parametric models in reinforcement learning? (2019, methodology): “rlo updates do not diverge, though they would be high variance. Third, we could employ algorithms speciﬁ- cally for stable off-policy learning, although these are often speciﬁc to the linear setting (Sutton et al., 2008, 2009; van Hasselt et al., 2014) or assume the sampling is done on trajectory (S”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “If seen in isolation, one might perhaps be tempted to think the observed instability is related to inherent instability problems of off-policy learning with function approximation (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2008; Maei, 2011; Sutton et al., 2015).”
Deep learning in neural networks: An overview (2014, background): “Wiering and Schmidhuber, 1998b; Baird and Moore, 1999; Meuleau et al., 1999; Morimoto and Doya, 2000; Bertsekas, 2001; Brafman and Tennenholtz, 2002; Abounadi et al., 2002; Lagoudakis and Parr, 2003; Sutton et al., 2008; Maei and Sutton, 2010; van Hasselt, 2012). Most are formulated in a probabilist”
Deep Reinforcement Learning and the Deadly Triad (2018, background): “Partial solutions have been proposed (Baird, 1995; Precup and Sutton, 2001; Sutton et al., 2008; Maei et al., 2009; Maei, 2011; Sutton et al., 2016), but they have mostly not been extended to non-linear ar X iv :1 81 2.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “The parameters or weights of C are trained to maximize reward by a standard RL method such as Q-learning or similar methods [17, 292, 293, 172, 254, 212, 262, 10, 122, 188, 157, 281, 26, 216, 197, 272, 311, 9, 163, 174, 22, 27, 2, 137, 276, 156, 284].”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “For each sample i in the training set D , we can calculate an unbiased estimate of A , b , and C as follows:”
GQ(lambda): A general gradient algorithm for temporal-difference prediction learning with eligibility traces (2010, methodology): “We use Lemma 6.7 (Bertsekas and Tsitsiklis 1996) that can be applied here and follow the proof of convergence for the TDC algorithm in Sutton et al. (2009b).”
Adaptive Control of Epileptic Seizures using Reinforcement Learning (2010, methodology): “More recently, a fast convergent off-policy TD(0) algorithm with linear function approximation, called GTD for gradient temporal-difference, was introduced to remove those limitations [66].”
Learning to Predict Independent of Span (2015, unknown): “…∑∞ t=0 α 2 t < ∞, and limt→∞ βt αt = 0, algorithm (44) converges almost surely to the TD fixed-point solution θ∗ that minimizes the mean-squared projected Bellman error (Sutton, Szepesvári, and Maei 2008; Sutton et al. 2009), such that E [ (Zt(θ∗)− φ>t θ∗)φ>t ] E[φtφt]−1 E [ φt(Zt(θ∗)− φ>t θ∗) ]…”
Scalable Bilinear π Learning Using State and Action Features (2018, methodology): “Linear basis functions have been used for both policy evaluation and optimization (Tsitsiklis and Van Roy, 1997; Nedić and Bertsekas, 2003; Lagoudakis and Parr, 2003; Melo et al., 2008; Sutton et al., 2009).”
Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus (2019, methodology): “, [5,6], f(w(t), θ(t)) represents the Actor update to improve the current policy, and M(t + 1) and N(t + 1) are sampling noises.”
Linear Programming for Large-Scale Markov Decision Problems (2014, background): “For a survey on theoretical results for ADP see (Bertsekas and Tsitsiklis, 1996, Sutton and Barto, 1998), (Bertsekas, 2007, Vol. 2, Chapter 6), and more recent papers (Sutton et al., 2009b,a, Maei et al., 2009, 2010).”