Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation

By Josiah P. Hanna, Peter Stone, Scott Niekum. 2016

95 citations recorded by the source corpus.

In many reinforcement learning applications, it is desirable to determine confidence interval lower bounds on the performance of any given policy without executing said policy. In this context, we propose two bootstrapping off-policy evaluation methods which use learned MDP transition models in order to estimate lower confidence bounds on policy performance with limited data. We empirically evaluate the proposed methods in a standard policy evaluation tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Software, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Software

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Alternatively, we can construct confidence bounds based on distributional assumptions (e.g., asymptotic normality) (Thomas et al., 2015b) or via bootstrapping (Thomas et al., 2015b; Hanna et al., 2017) which may be less conservative at the cost of looser guarantees.”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “Importance reweighting, or inverse propensity scoring, has been used for off-policy RL (e.g., Precup et al., 2001; Murphy et al., 2001; Munos et al., 2016; Hanna et al., 2017; Xie et al., 2019).”
Learning When-to-Treat Policies (2019, background): “In reinforcement learning, there has been extensive work focused on learning good models (Farajtabar et al., 2018; Hanna et al., 2017; Liu et al., 2018b).”
Representation Balancing MDPs for Off-Policy Policy Evaluation (2018, background): “[9] J.”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “Prior works [7, 10, 13, 21, 28, 34] do so with importance sampling (IS) [11], MDP modeling, or both.”
Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation (2020, methodology): “We are aware of one previous instance in which statistical bootstrapping was used for high-confidence policy evaluation with DM; specifically, [11] proposes to use Efron’s bootstrap in conjunction with model-based learning, similar to the present work.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “By contrast, probabilistically safe approaches aim to provide lower-bounds on expected return, rather than utilizing logical specifications of safety [387, 386, 179, 141, 385].”
Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, methodology): “While model-based policy evaluation has been considered by previous works (Paduraru, 2007; Thomas and Brunskill, 2016a; Hanna et al., 2017), it has largely been confined to simple domains with finite state and action spaces where function approximation is not necessary.”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, background): “Works fall into three main categories: importance sampling (IS) (e.g. Precup (2000); Jiang & Li (2015)), model-based (often referred to as the direct method) (e.g. Hanna et al. (2017); Gottesman et al. (2019b)), and value-based (e.g. Le et al. (2019)).”
Offline Policy Selection under Uncertainty (2020, background): “…there is also a growing body of literature that uses frequentist principles to derive high-conﬁdence lower bounds for the value of a policy (Bottou et al., 2013; Thomas et al., 2015b; Hanna et al., 2016; Kuzborskij et al., 2020; Feng et al., 2020; Dai et al., 2020; Kostrikov and Nachum, 2020).”