Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
By Aniruddh Raghu, Omer Gottesman, Yao Liu, Matthieu Komorowski, Aldo A. Faisal, Finale Doshi-Velez, Emma Brunskill. 2018
37 citations recorded by the source corpus.
In this work, we consider the problem of estimating a behaviour policy for use in Off-Policy Policy Evaluation (OPE) when the true behaviour policy is unknown. Via a series of empirical studies, we demonstrate how accurate OPE is strongly dependent on the calibration of estimated behaviour policy models: how precisely the behaviour policy is estimated from data. We show how powerful parametric models such as neural networks can result in highly uncalibrated behaviour policy models on a real-world medical dataset, and illustrate how a simple, non-parametric, k-nearest neighbours model produces better calibrated behaviour policy estimates and can be used to obtain superior importance sampling-based OPE estimates.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- Justin Fu
- George Tucker
- Pieter Abbeel
- Ilya Kostrikov
- Chelsea Finn
Citations in the corpus
- Reinforcement Learning in Healthcare: A Survey (2019, background): “should be defined in a way that to the most approximates the behavior policy that has generated such data [306], [307].”
- Benchmarks for Deep Off-Policy Evaluation (2021, background): “Batch (or offline) reinforcement learning, has been studied extensively in domains such as healthcare (Thapa et al., 2005; Raghu et al., 2018), recommender systems (Dudík et al., 2014; Theocharous et al., 2015; Swaminathan et al., 2017), education (Mandel et al., 2014), and robotics (Kalashnikov et…”
- POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, methodology): “In the real data setting, we estimate the behavior policy via the k-nearest neighbors approach of Raghu et al. (2018).”
- Model-Based Reinforcement Learning for Sepsis Treatment (2018, methodology): “We follow the suggestions of [16] when constructing these estimators.”