Fast gradient-descent methods for temporal-difference learning with linear function approximation

By Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvari, Eric Wiewiora. 2009

711 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Residual gradient (RG) and gradient temporal difference (GTD) methods were introduced in (Baird, 1995; Sutton et al., 2009a,b).”
When to use parametric models in reinforcement learning? (2019, methodology): “ methods (Sutton, 1988)), and off-policy learning (Sutton and Barto, 2018; Precup et al., 2000) can be unstable (Williams and Baird III, 1993; Baird, 1995; Sutton, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2009, 2016) — this is sometimes called the deadly triad (Sutton and Barto, 2018; van ”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “By repeatedly applying this recurrence, we can obtain the fixed point of the projected Bellman operator as k →∞ (Sutton et al., 2009).”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Further, fully gradient-based objectives for Q-learning [22, 23, 24] gave rise to convergent ADP algorithms with function approximation.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “Readers familiar with Gradient TD methods (Sutton et al., 2009a;b) or residual gradients (Baird, 1995) may ﬁnd the ﬁrst term of this objective similar to the Mean Squared Bell-man Error (MSBE).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “Typically, the Q -values are estimated by ﬁnding the ﬁxed point of the Bellman recurrence (Sutton et al., 2008, 2009; Scherrer, 2010; Liu et al., 2015; Dai et al., 2016; Du et al., 2017); i.e. , minimizing the (surrogate) squared diﬀerence between the LHS and RHS of (23), potentially with target…”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “Indeed, the motivation behind algorithms such as GTD (Sutton et al., 2009b;a) and residual methods (Baird, 1995; Scherrer, 2010) can be seen as introducing a stationary objective that can be optimized with standard procedures such as gradient descent for increased stability.”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “gence to the ﬁxed point with off-policy data occurs, solutions can be of poor quality. Thus, the existing TD learning algorithms with convergence guarantees under off-policy data (Maei et al. 2009), (Sutton et al. 2009) can still suffer from off-policy issues. This paper studies the covariate shift ”
Stochastic Primal-Dual Q-Learning (2018, neutral): “…RL, such as the distributed RL (Kar et al., 2013; Macua et al., 2015; Zhang et al., 2018; Lee et al., 2018), Q-learning with function approximations (Sutton et al., 2009b,a), sparsity promoted RL (Mahadevan and Liu, 2012), safe RL (Garcıa and Fern´andez, 2015), and the inverse RL (Ng and Russell,…”
Smoothed Dual Embedding Control (2017, background): “In recent years, a few authors have made important progress toward finding scalable, convergent TD algorithms, by designing proper objective functions and using stochastic gradient descent (SGD) to optimize them (Sutton et al., 2009; Maei, 2011).”
Emphatic TD Bellman Operator is a Contraction (2015, background): “In recent years, several different off-policy policy-evalu tion algorithms have been proposed and analyzed, such as importance-sampling based least-squareTD (Yu, 2012), gradient-based TD (Sutton et al., 2009), and ETD (Sutton et al., 2015).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “= Φ(Φ>MΦ)−1Φ>M ( T (λ)(Φθ)−Φθ ) (see Sutton et al. 2009) = Φ(Φ>MΦ)−1Φ>M ( (I−PπΓΛ)−1rπ + PλπΦθ −Φθ ) (by (29)) = Φ(Φ>MΦ)−1 ( b + Φ>M(Pλπ − I)Φθ ) (by (23)) = Φ(Φ>MΦ)−1 (b−Aθ) .”
Linear Off-Policy Actor-Critic (2012, methodology): “Gradient-TD methods (Sutton et al., 2009) incrementally learn the weights, v , in an oﬀ-policy setting, with a guarantee of stability and a linear per-time-step complexity.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “…of this article, broadly speaking we can categories OPE methods into groups based the use of importance sampling (Precup, 2000), value functions (Sutton et al., 2009; Migliavacca et al., 2010; Sutton et al., 2016; Yang et al., 2020), and learned transition models (Paduraru, 2007), though a…”