Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation

By Zhaohan Daniel Guo, Philip S. Thomas, Emma Brunskill. 2017

42 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, background): “Hence, they are dogged by high variance in the estimators common to IS, although recent work aims to address this (Munos et al., 2016; Guo et al., 2017).”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, methodology): “These methods use algorithms based on importance sampling (Precup, 2000; Thomas et al., 2015; Guo et al., 2017), action-value function approximation (model based) (Hallak et al., 2015) or doubly robust combination of both approaches (Jiang & Li, 2015).”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “A few authors explored the alternative to estimate the propensity score of a state’s stationary distribution [8, 21], when behavior policies are known.”
Batch Stationary Distribution Estimation (2020, background): “Standard importance weighting is known to have high variance, and various techniques have been proposed to reduce variance (Precup et al., 2001; Jiang and Li, 2016; Rubinstein and Kroese, 2016; Thomas and Brunskill, 2016; Guo et al., 2017).”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “A few variancereduction techniques have been introduced, but still cannot eliminate this fundamental issue (Jiang & Li, 2015; Thomas & Brunskill, 2016; Guo et al., 2017).”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”
Learning When-to-Treat Policies (2019, neutral): “We note that the closest works to ours are Jiang and Li (2016), Thomas and Brunskill (2016) and Zhang, Tsiatis, Laber, and Davidian (2013), with the key difference that our proposal has universal scores and nuisance components for all policies in a policy class, and thus is practical for policy opti”
Safe Policy Improvement with Baseline Bootstrapping (2017, background): “Indeed, Guo et al. (2017) show that the amount of data required by IS-based SPI algorithms scales exponentially with the horizon of the MDP.”
Batch Policy Learning under Constraints (2019, background): “In general, off-policy policy evaluation in longhorizon domains is known to be challenging (Liu et al., 2018; Guo et al., 2017).”
Representation Balancing MDPs for Off-Policy Policy Evaluation (2018, background): “Importance sampling (IS) based approaches [16, 23, 8, 5, 10, 22] are typically unbiased and strongly consistent, but despite recent progress tend to have high variance—especially if the evaluation policy is deterministic, as evaluating deterministic policies requires finding in the data”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “However, the variance of IS-based approaches [Precup et al., 2000; Thomas et al., 2015; Jiang and Li, 2016; Thomas and Brunskill, 2016; Guo et al., 2017; Farajtabar et al., 2018] tends to be too high to provide informative results, for long-horizon problems [Mandel et al., 2014], since the variance…”
Eigenoption Discovery through the Deep Successor Representation (2017, unknown): “In the reinforcement learning framework, options are a well-known formalization of the notion of actions extended in time; and they have been shown to speed up learning and planning when appropriately deﬁned ( e.g. , Brunskill & Li, 2014; Guo et al., 2017; Solway et al., 2014).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…importance sampling [Dud´ık et al., 2011, Mandel et al., 2014, Thomas et al., 2015, Li et al., 2015, Jiang and Li, 2016, Thomas and Brunskill, 2016, Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara…”
Universal Off-Policy Evaluation (2021, methodology): “Limitations: Our method uses importance sampling and thus (1) Requires knowledge of action probabilities under the behavior policy β, (2) Any outcome under the evaluation policy should have a sufficient probability of occurring under β, and (3) Variance of our estimators scales exponentially with th”
Optimizing for the Future in Non-Stationary MDPs (2020, background): “In particular, the use of importance sampling estimators can increase the variance further (Guo et al., 2017).”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, background): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”
Policy Optimization via Importance Sampling (2018, methodology): “A more theoretically grounded analysis has been provided for policy selection [10], model-free [55] and model-based [53] policy evaluation (also accounting for samples collected with multiple behavioral policies), and combined with options [14].”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, unknown): “…on this topic, see [Dud́ık et al., 2011, Mandel et al., 2014, Thomas et al., 2015, Li et al., 2015, Jiang and Li, 2016, Thomas and Brunskill, 2016, Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara…”