Off-policy TD( l) with a true online equivalence

By Hado van Hasselt, A. Mahmood, Richard S. Sutton. 2014

41 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “Third, we could employ algorithms specifically for stable off-policy learning, although these are often specific to the linear setting [Sutton et al., 2008, 2009, van Hasselt et al., 2014] or assume the sampling is done on trajectory [Sutton et al., 2016].”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “…which includes the original linear-time algorithm (Sutton, 1988), least-squares formulations (Bradtke & Barto, 1996; Mahmood et al., 2014), methods for adapting λ (Downey & Sanner, 2010), true-online methods (van Hasselt et al., 2014), and the recent emphatic methods (Mahmood et al., 2015).”
Emphatic Temporal-Difference Learning (2015, background): “The importance-sampled λ-return for off-policy learning is thus defined as follows (Maei 2011, van Hasselt et al. 2014): Gλρt .”
True Online Temporal-Difference Learning (2015, background): “Van Hasselt et al. (2014) took the approach of providing a theorem that proves equivalence between a general forward view and a corresponding general backward view.”
Convergent Tree-Backup and Retrace with Function Approximation (2017, methodology): “True on-line equivalence: In van Hasselt et al. (2014), the authors derived a true on-line update for GTD(λ) that empirically performed better than GTD(λ) with eligibility traces. Based on this work, we derive true on-line updates for our algorithm. The gradient off-policy algorithm was derived by t”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, background): “…environment and in that sense are model-free (Precup et al. 2000, 2001, Maei & Sutton 2010, Maei 2011, Yu 2012, Geist & Scherrer 2014, Dann et al. 2014, van Hasselt et al. 2014, Mahmood et al. 2014, Yu 2015, 2016, Hallak et al. 2015a, Mahmood et al. 2015, Sutton et al. 2016, White & White 2016a).”
Investigating Practical Linear Temporal Difference Learning (2016, methodology): “where gw,k is called xk in van Hasselt’s Theorem 1 [23].”
Reinforcement learning improves behaviour from evaluative feedback (2015, background): “This insight was rapidly generalized to non-linear function approxi-mators that are smooth (locally linear) 18 , to the control setting 19 , and to the use of eligibility traces in the learning process 20 .”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, background): “Additionally, there have been a variety of algorithms derived for the linear setting, both for on-policy and off-policy learning (Sutton et al., 2009; Maei, 2011; van Seijen and Sutton, 2014; van Hasselt et al., 2014; Mahadevan et al., 2014; Sutton et al., 2016; Mahmood et al., 2017).”
True Online Emphatic TD(λ): Quick Reference and Implementation Guide (2015, methodology): “Here we present these ideas without assuming access to an underlying state (as in Modayil, White & Sutton 2014).”
Off-policy learning based on weighted importance sampling with linear computational complexity (2015, methodology): “We propose two new off-policy algorithms: WIS-GTD( � ) and WIS-TO-GTD( � ) , based on GTD( � ) (Maei 2011) and true online GTD( � ) (van Hasselt et al. 2014), respectively.”
Learning to Predict Independent of Span (2015, background): “(5) An eligibility trace of this special form is called a dutch trace (van Hasselt, Mahmood, and Sutton 2014).”
Multiagent Fully Decentralized Value Function Learning With Linear Convergence Rates (2018, methodology): “There exist a plethora of algorithms for policy evaluation such as GTD [5], TDC [6], GTD2 [6], GTD-MP/GTD2-MP [7], GTD(λ) [8], and True Online GTD(λ) [9].”