Universal Off-Policy Evaluation

By Yash Chandak, Scott Niekum, Bruno Castro da Silva, Eric Learned-Miller, Emma Brunskill, Philip S. Thomas. 2021

61 citations recorded by the source corpus.

When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO) -- one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss Uno's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “As a result, replay has become a very successful approach for tackling continual RL (Isele and Cosgun, 2018; Riemer et al., 2019; Rolnick et al., 2019; Oh et al., 2021; Henning et al., 2021; Liu et al., 2021b; Queeney et al., 2021; Chandak et al., 2021; Lampinen et al., 2021; Venuto et al., 2021; Li”