Doubly Robust Off-policy Evaluation for Reinforcement Learning

By Nan Jiang, Lihong Li. 2015

67 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep-Treat: Learning Optimal Personalized Treatments From Observational Data Using Neural Networks (2018, methodology): “The doubly robust estimation techniques combine the direct and IPS methods and generate more robust counterfactual estimates (Dudı́k, Langford, and Li 2011; Jiang and Li 2016).”
Combining Kernel and Model Based Learning for HIV Therapy Selection (2017, background): “• Doubly Robust Evaluation [21] further reduces variance and acheives consistency by coupling the importance score with a regression estimate of the value of the treatment policy.”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “We also compare to the guided importance sampling doubly robust (DR) estimator (Jiang & Li, 2015).”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “Importance reweighting, or inverse propensity scoring, has been used for oﬀ-policy RL [12,18,28,29,32,39].”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “…robustness for off-policy evaluation, orthogonal to the existing literature of doubly robust policy evaluation that solely devotes to variance reduction (Jiang & Li, 2016; Thomas & Brunskill, 2016; Farajtabar et al., 2018), mostly based on the idea of control variates (e.g. Asmussen & Glynn, 2007).”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, methodology): “When extended to long-horizon problems, it faces an exponential blowup of variance, and variance-reduction techniques are used to improve the estimator [10, 14, 39, 42].”
Deep Reinforcement Learning for Sepsis Treatment (2017, methodology): “See [3] for more information.”
Optimal and Adaptive Off-policy Evaluation in Contextual Bandits (2016, methodology): “Throughout this section, we drop the superscript π from value estimators, as the evaluation policy π is ﬁxed throughout this discussion.”
Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach (2017, methodology): “Furthermore, directly comparing Q values on off-policy data, as done in prior applications of RL to healthcare (Komorowski et al., 2016) can provide incorrect performance estimates (Jiang and Li, 2015).”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, methodology): “Despite being noted in the literature to suffer from high variance, importance sampling is a common component of many policy evaluation algorithms [22, 8].”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, methodology): “In order for importance sampling to provide unbiased estimates, we require the assumption that supp(µi) ⊆ supp(µb) for all i ∈ {1, . . . , l}, which is standard in off-policy policy evaluation research [Precup et al., 2000; Thomas et al., 2015a; Jiang and Li, 2015].”
Learning Optimal Policies from Observational Data (2018, methodology): “The off-policy evaluation methods include IPS estimator (Rosenbaum & Rubin, 1983; Strehl et al., 2010), self normalizing estimator (Swaminathan & Joachims, 2015b), direct estimation, doubly robust estimator (Dud´ık et al., 2011; Jiang & Li, 2016) and matching based methods (Hill & Reiter, 2006).”
High Confidence Off-Policy Evaluation with Models (2016, methodology): “Model bias in MDPs can come from two sources: 1.”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, methodology): “Due to performing model estimation in addition to value function estimation, these algorithms tend to be computationally more complex (Dud´ık et al. 2011, Paduraru 2013, Hallak et al. 2015b, Li et al. 2015, Jiang & Li 2015, Thomas & Brunskill 2016).”
A Bayesian Nonparametic Approach for Estimating Individualized Treatment-Response Curves (2016, background): “For example, doubly-robust estimators for policy evaluation have been developed for contextual bandits (Dudik et al., 2011) and for sequential decision-making problems (Jiang and Li, 2015).”
Preventing undesirable behavior of intelligent machines (2019, methodology): “This would provide a more expressive policy class and would result in a more conventional RL formulation that allows for the use of more advanced off-policy evaluation techniques than the basic importance sampling estimator that we used [149, 195].”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “…robustness for off-policy evaluation, orthogonal to the existing literature of doubly robust policy evaluation that solely devotes to variance reduction (Jiang & Li, 2016; Thomas & Brunskill, 2016; Farajtabar et al., 2018), mostly based on the idea of control variates (e.g. Asmussen & Glynn, 2007).”
Constructing effective personalized policies using counterfactual inference from biased data sets with many features (2016, methodology): “The doubly robust methods compute the counterfactuals by combining direct and inverse propensity score reweighing methods to compute more robust estimates (Dudík et al. 2011; Jiang and Li 2016).”