Weighted importance sampling for off-policy learning with linear function approximation

By A. Mahmood, Hado van Hasselt, Richard S. Sutton. 2014

171 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “…example, Q-Prop can also be used with other on-policy policy gradient methods such as A3C (Mnih et al., 2016) and off-policy advantage estimation meth-ods such as Retrace( λ ) (Munos et al., 2016), GTD2 (Sutton et al., 2009), emphatic TD (Sutton et al., 2015), and WIS-LSTD (Mahmood et al., 2014).”
Prioritized Experience Replay (2015, methodology): “These weights can be folded into the Q-learning update by using w i δ i instead of δ i (this is thus weighted IS, not ordinary IS, see e.g. Mahmood et al., 2014).”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “Here we provide only an overview.”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “Lastly, there are many methods that combine on-and off-policy data for policy evaluation (Precup, 2000; Mahmood et al., 2014; Munos et al., 2016), mostly through variants of importance sampling.”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, background): “…policy, for a sequential decision-making problem using data generated by the log of another policy, known as the behavior policy (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016; Xie et al., 2019).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “Off-policy Policy Evaluation The problem of off-policy policy evaluation has been heavily studied in contextual bandits [8, 42, 45] and in the more general RL setting [10, 16, 20, 23, 28, 29, 30, 43, 44].”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, background): “[10] A.”
On Convergence of Emphatic Temporal-Difference Learning (2015, background): “For a recent work in this direction, see (Mahmood et al., 2014).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Parametric temporal-difference learning was first studied as the key “learning by generalization” algorithm in Samuel’s (1959) checker player.”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, background): “OPE is a key problem in reinforcement learning (RL) (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016) and it finds applications as varied as healthcare (Murphy, 2003) and education (Mandel et al.”
Emphatic Temporal-Difference Learning (2015, methodology): “An extension to variance-reduction methods, such as weighted importance sampling (Precup et al. 2000, Mahmood et al. 2014, 2015), can be a natural remedy to this problem.”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “Several variants have been proposed, such as step-wise IS and weighted IS [23].”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, methodology): “However, beyond simply constraining the iterates, more variance reduction techniques are needed, such as control variates [38, 1] and weighted importance sampling [36, 35, 27, 26].”
What is the Effect of Importance Weighting in Deep Learning? (2018, methodology): “The technique is also frequently employed in oﬀ-policy reinforcement learning [Precup, 2000, Mahmood et al., 2014, Swaminathan and Joachims, 2015], where we desire to learn a new policy given oﬄine samples collected from a preexisting policy.”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, background): “…environment and in that sense are model-free (Precup et al. 2000, 2001, Maei & Sutton 2010, Maei 2011, Yu 2012, Geist & Scherrer 2014, Dann et al. 2014, van Hasselt et al. 2014, Mahmood et al. 2014, Yu 2015, 2016, Hallak et al. 2015a, Mahmood et al. 2015, Sutton et al. 2016, White & White 2016a).”
Learning to Predict Independent of Span (2015, background): “…and, ultimately, optimal policy) are learnable through a proper use of rejection sampling, as in Q-learning, or importance sampling (Precup, Sutton, and Singh 2000; Precup and Sutton 2001; Maei 2011; Sutton et al. 2014; van Hasselt, Mahmood, and Sutton 2014; Mahmood, van Hasselt, and Sutton 2014).”
On the Role of Weight Sharing During Deep Option Learning (2019, background): “…these problems have been addressed in the literature by leveraging importance sampling (Precup, 2000b; Peshkin & Shelton, 2002; Jie & Abbeel, 2010; Mahmood et al., 2014) and scalable storage mechanisms (Robins, 1995; Li & Hoiem, 2016; Riemer et al., 2016, 2017; Shin et al., 2017; Riemer et al.,…”
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning (2019, methodology): “In that work, a distributed off-policy actor-critic algorithm is proposed that uses consensus for the actor updates – unlike [28,29] and our current work, which both use consensus for the critic step – and gradient temporal differencing (GTD) [14,19,26] for the critic updates, whereas the algorithm ”