High Confidence Off-Policy Evaluation with Models
By Josiah P. Hanna, Peter Stone, Scott Niekum. 2016
5 citations recorded by the source corpus.
In many reinforcement learning applications executing a poor policy may be costly or even dangerous. Thus, it is desirable to determine confidence interval lower bounds on the performance of any given policy without executing said policy. Current methods for high confidence off-policy evaluation require a substantial amount of data to achieve a tight lower bound, while existing model-based methods only address the problem in discrete state spaces. We propose two bootstrapping approaches combined with learned MDP transition models in order to efficiently estimate lower confidence bounds on policy performance with limited data in both continuous and discrete state spaces. Since direct use of a model may introduce bias, we derive a theoretical upper bound on model bias when we estimate the model transitions with i.i.d. sampled trajectories. This bound can be used to guide selection between the two methods. Finally, we empirically validate the data-efficiency of our proposed methods across three domains and analyze the settings where one method is preferable to the other.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yinlam Chow
- Mohammad Ghavamzadeh
- Mehrdad Farajtabar
- Masatoshi Uehara
- Ziyang Tang
- Nathan Kallus
- Ofir Nachum
- Sergey Levine
In Electrical and Electronic Engineering
Citations in the corpus
- More Robust Doubly Robust Off-policy Evaluation (2018, methodology): “, 2016) and has been used in some OPE applications (Hanna et al., 2016).”