High-Confidence Off-Policy Evaluation

By Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh. 2015

320 citations recorded by the source corpus.

Many reinforcement learning algorithms use trajectories collected from the execution of one or more policies to propose a new policy. Because execution of a bad policy can be costly or dangerous, techniques for evaluating the performance of the new policy without requiring its execution have been of recent interest in industry. Such off-policy evaluation methods, which estimate the performance of a policy using trajectories collected from the execution of other policies, heretofore have not provided confidences regarding the accuracy of their estimates. In this paper we propose an off-policy method for computing a lower confidence bound on the expected return of a policy.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “Another line of work relevant to risk sensitivity uses off-policy estimation to perform a policy update that is good with high probability [154].”
Deep Reinforcement Learning that Matters (2017, background): “Several works have begun investigating policy evaluation methods for the purposes of safe RL (Thomas and Brunskill 2016; Thomas, Theocharous, and Ghavamzadeh 2015), but further work is needed in signiﬁcance testing and statistical analysis.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Thomas et al. (2015b) derived confidence bounds based on concentration inequalities specialized to deal with the high variance and potentially large range of the importance weighted estimators.”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, background): “, 2014), and there has been work to mitigate the variance of IS estimators (Thomas et al., 2015; Thomas, 2015), the produced estimates are typically too high variance to be informative in long horizon settings with limited data.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “In practice, however, strict CIs are usually too pessimistic, and normal approximations are used instead (Thomas et al., 2015b).”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, methodology): “These methods use algorithms based on importance sampling (Precup, 2000; Thomas et al., 2015; Guo et al., 2017), action-value function approximation (model based) (Hallak et al., 2015) or doubly robust combination of both approaches (Jiang & Li, 2015).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, background): “The problem of off-policy policy evaluation has been heavily studied in contextual bandits [12, 49, 52] and in the more general RL setting [14, 21, 26, 29, 34, 36, 37, 50, 51].”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “…Dudı́k et al. 2011; Swaminathan et al. 2017) and reinforcement learning (RL) (e.g., Precup et al. 2000a; 2001; Paduraru 2013; Mahmood et al. 2014; Thomas et al. 2015a; Li et al. 2015; Jiang & Li 2016; Thomas & Brunskill 2016), and some of the results have been applied to problems in marketing…”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “In addition to the SPIBB algorithms, our ﬁnite MDP benchmark contains four algorithms: Basic RL, HCPI (Thomas et al., 2015a), Robust MDP, and RaMDP (Petrik et al., 2016).”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “[29] rely on the assumption that the importance sampling estimate is normally distributed which is false when the number of trajectories is small.”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “This is critical for several reasons, including providing high-confidence guarantees prior to deployment (Thomas et al., 2015), and performing policy improvement and model selection (Bottou et al.”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “A number of methods have also been suggested to reduce sample complexity when provided with either a baseline policy [25, 26], expert demonstration [27, 28], or approximate simulator [29, 30].”
Representation Balancing MDPs for Off-Policy Policy Evaluation (2018, background): “Importance sampling (IS) based approaches [16, 23, 7, 5, 9, 22] are typically unbiased 26 and strongly consistent, but despite recent progress tend to have high variance—especially if the 27 evaluation policy is deterministic, as evaluating deterministic policies requires finding in the data 28 sequ”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “However, the variance of IS-based approaches [Precup et al., 2000; Thomas et al., 2015; Jiang and Li, 2016; Thomas and Brunskill, 2016; Guo et al., 2017; Farajtabar et al., 2018] tends to be too high to provide informative results, for long-horizon problems [Mandel et al., 2014], since the variance…”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “[36] Thomas, P.”
Offline Evaluation of Online Reinforcement Learning Algorithms (2016, background): “These problems have ignited a recent renewal of interest in ofﬂine policy evaluation in the RL community (Mandel et al. 2014; Thomas, Theocharous, and Ghavamzadeh 2015), where one uses a precollected dataset to achieve high-quality estimates of the performance of a proposed policy.”
Personalized Ad Recommendation Systems for Life-Time Value Optimization with Guarantees (2015, methodology): “In this paper, we use our recently proposed model-free approach that computes a lower-bound on the expected return of a policy using a concentration inequality [Thomas et al., 2015a] to tackle the off-policy evaluation problem.”