Off-Policy Temporal Difference Learning with Function Approximation

By Doina Precup, Richard S. Sutton, Sanjoy Dasgupta. 2001

440 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Dynamic programming methods based on Q-function learning can also leverage off-policy data (Precup et al., 2001; Mnih et al., 2015; Lillicrap et al., 2016; Gu et al., 2016; Haarnoja et al., 2018b).”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “…RL, with and without function approximation, have been presented under the name of “off-policy” or “counterfactual” learning (see Precup et al. (2000), Precup et al. (2001), Bottou et al. (2013), Swaminathan and Joachims (2015a), Swaminathan and Joachims (2015b), Jiang and Li (2016), Thomas and…”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “These methods have been shown to be an effective approach, through importance sampling (Precup et al., 2001; Munos et al., 2016), distributed methods (Mnih et al.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Naive importance correction is hardly able to overcome these shortcomings in practice [28, 29].”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Related previous work on off-policy RL include (Precup et al., 2000; 2001; Wawrzynski, 2009; Geist & Scherrer, 2014) and (Harutyunyan et al., 2016).”
Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “T π Q π = Q π = ( I − γP π ) − 1 r .”
A Deep Reinforcement Learning Chatbot (2017, methodology): “The importance ratio c dt is known to exhibit very high, possibly inﬁnite, variance (Precup et al. 2001).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “)}i=1 are n trajectory samples from πβ(τ) (Precup, 2000; Precup et al., 2001; Peshkin and Shelton, 2002).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Earlier works [51, 49, 52] used per-action importance sampling on Monte-Carlo returns to obtain an OPE return estimator.”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…and importance sampling: Importance-sampling has been extensively explored for off-policy policy evaluation (Sutton & Barto, 2018; Precup, 2000; Precup et al., 2001; Munos et al., 2016; Espeholt et al., 2018; Nachum et al., 2018) as well as for counter-factual policy evaluation (Buesing et…”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Additionally, off-policy algorithms which rely on importance sampling (Precup et al., 2001; Jiang & Li, 2016; Munos et al., 2016) may not be applicable in a batch setting, requiring access to the action probabilities under the behavioral policy, and scale poorly to multi-dimensional action spaces.”
Self-Imitation Learning (2018, methodology): “Many existing methods are based on off-policy policy evaluation (Precup et al., 2001; 2000), which involves importance sampling.”
MOPO: Model-based Offline Policy Optimization (2020, background): “While many off-policy RL algorithms [52, 10, 30, 47, 42, 26, 19, 23, 24] can in principle be applied to a batch ofﬂine setting, they perform poorly in practice [18] due to poor extrapolation to out-of-distribution actions.”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “However, these methods typically have a variance that grows exponentially with the horizon, limiting their application to mostly short-horizon problems like contextual bandits [2,5].”
Batch Stationary Distribution Estimation (2020, methodology): “One important application of VPM is off-policy RL [Precup et al., 2001].”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “…sampling (IS) has has been explored in the contextual bandits (Strehl et al., 2010; Dudı́k et al., 2011; Wang et al., 2017), and episodic RL settings (Murphy et al., 2001; Precup et al., 2001), achieving many empirical successes (e.g., Strehl et al., 2010; Dudı́k et al., 2011; Bottou et al., 2013).”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, background): “A key challenge in reinforcement learning (RL, Sutton & Barto 2018) is off-policy evaluation (Precup et al., 2001; Maei, 2011; Jiang & Li, 2015; Sutton & Barto, 2018; Liu et al., 2018; Nachum et al., 2019; Zhang et al., 2020a), where we want to estimate the performance of a target policy (average re”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “…more typical to assume access to the distribution d π (for example, by simply performing Monte-Carlo rollouts enabled by online access), although instances exist in which d π is approximated by importance-weighting a diﬀerent ( i.e. , oﬄine) distribution (Precup et al., 2001; Sutton et al., 2014).”