Offline Evaluation of Online Reinforcement Learning Algorithms

By Travis Mandel, Yun-En Liu, Emma Brunskill, Zoran Popovic. 2016

22 citations recorded by the source corpus.

In many real-world reinforcement learning problems, we have access to an existing dataset and would like to use it to evaluate various learning approaches. Typically, one would prefer not to deploy a fixed policy, but rather an algorithm that learns to improve its behavior as it gains more experience. Therefore, we seek to evaluate how a proposed algorithm learns in our environment, meaning we need to evaluate how an algorithm would have gathered experience if it were run online. In this work, we develop three new evaluation approaches which guarantee that, given some history, algorithms are fed samples from the distribution that they would have encountered if they were run online. Additionally, we are the first to propose an approach that is provably unbiased given finite data, eliminating bias due to the length of the evaluation. Finally, we compare the sample-efficiency of these approaches on multiple datasets, including one from a real-world deployment of an educational game.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, methodology): “In this view, evaluations can use the average cumulative rewards across the entire learning process (balancing exploration and exploitation) as in (Hofer and Gimbert 2016), or can possibly use offline evaluation as in (Mandel et al. 2016).”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “More recent methods can be found in the works of Jiang & Li (2015) and Mandel et al. (2016).”
More Robust Doubly Robust Off-policy Evaluation (2018, background): “…et al. 2015a; Li et al. 2015; Jiang & Li 2016; Thomas & Brunskill 2016), and some of the results have been applied to problems in marketing (e.g., Li et al. 2011; Theocharous et al. 2015), healthcare (e.g., Murphy et al. 2001; Hirano et al. 2003), and education (e.g., Mandel et al. 2014; 2016).”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, methodology): “[12] introduce several methods for evaluating on-line reinforcement learning algorithms with offline data.”