Importance Sampling Policy Evaluation with an Estimated Behavior Policy

By Josiah P. Hanna, Scott Niekum, Peter Stone. 2018

73 citations recorded by the source corpus.

We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account for differences in the likelihood of the returns between the two policies. In this paper, we study importance sampling with an estimated behavior policy where the behavior policy estimate comes from the same set of data used to compute the importance sampling estimate. We find that this estimator often lowers the mean squared error of off-policy evaluation compared to importance sampling with the true behavior policy or using a behavior policy that is estimated from a separate data set. Intuitively, estimating the behavior policy in this way corrects for error due to sampling in the action-space. Our empirical results also extend to other popular variants of importance sampling and show that estimating a non-Markovian behavior policy can further lower large-sample mean squared error even when the true behavior policy is Markovian.

Subfields: Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “Furthermore, IS explicitly uses the knowledge of πb, and when it is not available the policy has to be estimated from data (Hanna et al., 2019).”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “, 2017) and more general finite horizon RL settings (Fonteneau et al., 2013; Li et al., 2015; Jiang & Li, 2016; Thomas & Brunskill, 2016; Liu et al., 2018b; Farajtabar et al., 2018; Hanna et al., 2019; Xie et al., 2019).”
Safe Policy Improvement with an Estimated Baseline Policy (2019, neutral): “Hanna, Niekum, and Stone (2019) reported similar results in a different setting and formally explained them with the sampling error.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Since the behavior policy is not known explicitly, we learn an estimate of it via a max-likelihood objective over the dataset D , as advocated by Xie et al. (2018); Hanna et al. (2019).”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “IPS often assumes known πb, which may not be possible – one approach is to estimate πb from data (Hanna et al., 2019), resulting in a potentially biased estimator that can sometimes outperform traditional IPS methods.”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, background): “…has been studied in contextual bandits (Dudík et al., 2011; Wang et al., 2017) and more general ﬁnite horizon RL settings (Fonteneau et al., 2013; Li et al., 2015; Jiang & Li, 2016; Thomas & Brunskill, 2016; Liu et al., 2018b; Farajtabar et al., 2018; Hanna et al., 2019; Xie et al., 2019).”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “…to MDPs (Ghavamzadeh et al., 2016; Petrik & Russell, 2019), us-ing model-based RL to safely improve upon suboptimal demonstrations (Thananjeyan et al., 2019), and obtaining high-conﬁdence off-policy bounds on the performance of an evaluation policy (Thomas et al., 2015; Hanna et al., 2019).”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “Since the behavior policy is not known explicitly, we learn an estimate of it via a max-likelihood objective over the dataset D , as advocated by Hanna et al. (2019).”