Off-policy learning based on weighted importance sampling with linear computational complexity

By A. Mahmood, Richard S. Sutton. 2015

32 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Q(λ) with Off-Policy Corrections (2016, background): “There are several (recent) off-policy algorithms that reduce the variance of IS methods, at the cost of added bias [5, 6, 3].”
Safe and Efficient Off-Policy Reinforcement Learning (2016, methodology): “It is well known that IS estimates can suffer from large – even possibly infinite – variance (mainly due to the variance of the product π(a1|x1) μ(a1|x1) · · · π(at|xt) μ(at|xt) ), which has motivated further variance reduction techniques such as in (Mahmood and Sutton, 2015; Mahmood et al., 2015; H”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, methodology): “However, beyond simply constraining the iterates, more variance reduction techniques are needed, such as control variates [38, 1] and weighted importance sampling [36, 35, 27, 26].”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, background): “Unfortunately, the problem of large variance with importance sampling is also the most severe in multi-step learning (White 2015, Mahmood & Sutton 2015).”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, methodology): “These linear methods have also been well-explored theoretically (Tsitsiklis and Van Roy, 1997; Maei, 2011; Mahmood and Sutton, 2015; Yu, 2015) and empirically (Dann et al., 2014; White and White, 2016), with some insights into improvements from gradient methods (Sutton et al., 2009), true-online…”
True Online Temporal-Difference Learning (2015, background): “Recently, Mahmood & Sutton (2015) extended this theorem further by proving equivalence between an even more general forward view and backward view.”
Q( λ ) with Oﬀ-Policy Corrections (background): “There are several (recent) oﬀ-policy algorithms that reduce the variance of IS methods, at the cost of added bias [5, 6, 3].”
Incremental Off-policy Reinforcement Learning Algorithms (2017, background): “Chapter 7 is based on a published paper (Mahmood & Sutton 2015) which resulted from our effort to extend the above work to linear computational complexity.”
Investigating Practical Linear Temporal Difference Learning (2016, methodology): “Following the general expected update in (4), ht+1 ← ht + αh ( (−Aπwt + bπ)−A>µ ht ) (7) which converges if A>µ is positive definite for any fixed wt and αh is chosen appropriately (see Sutton et al.’s recent paper[21] for an extensive discussion of convergence in expectation).”