A Convergent O(n) Temporal-difference Algorithm for Off-policy Learning with Linear Function Approximation

By Richard S. Sutton, Csaba Szepesvari, Hamid Reza Maei. 2008

277 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Convergent backups [47, 46, 32] were developed.”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “Readers familiar with Gradient TD methods (Sutton et al., 2009a;b) or residual gradients (Baird, 1995) may ﬁnd the ﬁrst term of this objective similar to the Mean Squared Bell-man Error (MSBE).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “Typically, the Q -values are estimated by ﬁnding the ﬁxed point of the Bellman recurrence (Sutton et al., 2008, 2009; Scherrer, 2010; Liu et al., 2015; Dai et al., 2016; Du et al., 2017); i.e. , minimizing the (surrogate) squared diﬀerence between the LHS and RHS of (23), potentially with target…”
Generalized Off-Policy Actor-Critic (2019, methodology): “To approach this issue, Gradient TD methods (Sutton et al. 2009) were proposed.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “Indeed, the motivation behind algorithms such as GTD (Sutton et al., 2009b;a) and residual methods (Baird, 1995; Scherrer, 2010) can be seen as introducing a stationary objective that can be optimized with standard procedures such as gradient descent for increased stability.”
Stochastic Primal-Dual Q-Learning (2018, neutral): “…such as the distributed RL (Kar et al., 2013; Macua et al., 2015; Zhang et al., 2018; Lee et al., 2018), Q-learning with function approximations (Sutton et al., 2009b,a), sparsity promoted RL (Mahadevan and Liu, 2012), safe RL (Garcıa and Fern´andez, 2015), and the inverse RL (Ng and Russell,…”
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search (2019, methodology): “Several techniques have been proposed to mitigate this problem, such as the entropy measures in Soft Actor-Critic[21] and Proximal Policy Optimization [48], adding noise in Deep Deterministic Policy Gradient [29], and the experience replay in Off-Policy Policy Gradient [55].”
Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTD (2018, methodology): “INTRODUCTION The goal of this paper is to study distributed version of the gradient temporal-difference (GTD) learning algorithm, originally presented in [1], [2], for multi-agent Markov decision processes (MDPs).”
Linear Stochastic Approximation: How Far Does Constant Step-Size and Iterate Averaging Go? (2018, methodology): “Error bounds for GTD/TD The initial convergence analysis for GTD/GTD2/TDC was only asymptotic in nature [21, 20] with diminishing stepsizes.”
Sample-Optimal Parametric Q-Learning Using Linearly Additive Features (2019, background): “Sutton et al. [2009] shows off-policy temporal-difference learning is convergent with linear function approximation.”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “In order to learn off-policy, this mismatch can be corrected using a variety of techniques [43, 25, 35, 30].”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “In general, off-policy algorithms need not estimate the optimal policy, but can also approximate an arbitrary other policy (Precup et al, 2000; Precup and Sutton, 2001; Sutton et al, 2008; van Hasselt, 2011, Section 5.4).”
Generalized TD Learning (2011, methodology): “…that TD-based algorithms (TD Sutton and Barto, 1998, NTD Bradtke and Barto, 1996, LSTD Bradtke and Barto, 1996, LSPE Nedić and Bertsekas, 2003, GTD Sutton et al., 2009b, GTD2, TDC Sutton et al., 2009a and iLSTD Geramifard et al., 2006), TD (λ)-based algorithms (TD (λ) Sutton and Barto, 1998, NTD…”
CrossNorm: Normalization for Off-Policy TD Reinforcement Learning (2019, methodology): “GTD-2 and TD with gradient Correction (TDC) (Sutton et al., 2009a) also come with convergence guarantees for non-linear function approximators.”