Eligibility Traces for Off-Policy Policy Evaluation

By Doina Precup, Richard S. Sutton, Satinder Singh. 2000

933 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “(We also experimented with the related tree backup method of Precup et al. (2000) but found Retrace to perform better in practice.)”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[11, 29, 45, 60]).”
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “Our contribution is to modify the self-normalizing estimator (Kong, 1992; Precup et al., 2000; Swaminathan and Joachims, 2015b; Joachims et al., 2018) to be applicable to neural networks.”
Can Neural Machine Translation be Improved with User Feedback? (2018, background): “, 2013) and off-policy reinforcement learning (Precup et al., 2000; Jiang and Li, 2016) can be used to avoid degenerate behavior of estimators under deterministic logging.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…to explore and collect more data (Degris et al., 2012; Ried-miller, 2005) such as learning online from an out-dated replay buffer (e.g. (Munos et al., 2016)), or have performed off-policy policy evaluation (Fara-jtabar et al., 2018; Jiang and Li, 2016; Precup, 2000; Thomas and Brunskill, 2016).”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Offline methods in RL, with and without function approximation, have been presented under the name of “off-policy” or “counterfactual” learning (see Precup et al. (2000), Precup et al. (2001), Bottou et al. (2013), Swaminathan and Joachims (2015a), Swaminathan and Joachims (2015b), Jiang and Li…”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Naive importance correction is hardly able to overcome these shortcomings in practice [28, 29].”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “The updates for this method are given below, where π θ ( a | ) is a deterministic policy, β is arbitrary exploration distribution, and ρ β corresponds to sampling from a replay buffer.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Tree Backup (Precup et al., 2000), and Q(λ) (Sutton et al.”
When to use parametric models in reinforcement learning? (2019, methodology): “, neural networks), bootstrapping (as in temporal difference methods (Sutton, 1988)), and off-policy learning (Sutton and Barto, 2018; Precup et al., 2000) can be unstable (Williams and Baird III, 1993; Baird, 1995; Sutton, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Related previous work on off-policy RL include (Precup et al., 2000; 2001; Wawrzynski, 2009; Geist & Scherrer, 2014) and (Harutyunyan et al., 2016).”
Neural Approaches to Conversational AI (2018, neutral): “real human-human conversation corpora directly for evaluation. Unfortunately, this problem, known as off-policy evaluation in the RL literature, is challenging with numerous current research efforts (Precup et al., 2000; Jiang and Li, 2016; Thomas and Brunskill, 2016; Liu et al., 2018c). Such off-po”
Safe and Efficient Off-Policy Reinforcement Learning (2016, methodology): “Importance sampling is the simplest way to correct for the discrepancy between µ and π when learning from off-policy returns (Precup et al., 2000, 2001; Geist and Scherrer, 2014).”
A Deep Reinforcement Learning Chatbot (2017, methodology): “Off-policy Reinforcement Learning : We use a variant of the classical REINFORCE algo-rithm (Williams 1992, Precup 2000, Precup et al. 2001), which we call Off-policy REINFORCE .”
SNAS: Stochastic Neural Architecture Search (2018, background): “To assign credits to actions both temporally and laterally is an important topic in reinforcement learning (Precup, 2000; Schulman et al., 2015; Tucker et al., 2018; Xu et al., 2018).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…b(sit)), then because rt does not depend on st′ and at′ for t′ > t, we can drop importance weights in the future, resulting in the per-decision importance sampling policy gradient estimator (Precup, 2000): ∇θJ(πθ) ≈ n∑ i=1 H∑ t=0 witγ t ( H∑ t′=t γt ′−tw i t′ wit rt′ − b(sit) ) ∇θ log πθ(ait|sit).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Earlier works [51, 49, 52] used per-action importance sampling on Monte-Carlo returns to obtain an OPE return estimator.”
Meta-Gradient Reinforcement Learning (2018, background): “The n -step or λ -return can be augmented with off-policy corrections [Precup et al., 2000, Sutton et al., 2014, Espeholt et al., 2018] if it is necessary to correct for the distribution used to generate the data.”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “…and importance sampling: Importance-sampling has been extensively explored for off-policy policy evaluation (Sutton & Barto, 2018; Precup, 2000; Precup et al., 2001; Munos et al., 2016; Espeholt et al., 2018; Nachum et al., 2018) as well as for counter-factual policy evaluation…”