Offline Policy Selection under Uncertainty

By Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, Dale Schuurmans. 2020

35 citations recorded by the source corpus.

The presence of uncertainty in policy evaluation significantly complicates the process of policy ranking and selection in real-world settings. We formally consider offline policy selection as learning preferences over a set of policy prospects given a fixed experience dataset. While one can select or rank policies based on point estimates of their policy values or high-confidence intervals, access to the full distribution over one's belief of the policy value enables more flexible selection algorithms under a wider range of downstream evaluation metrics. We propose BayesDICE for estimating this belief distribution in terms of posteriors of distribution correction ratios derived from stochastic constraints (as opposed to explicit likelihood, which is not available). Empirically, BayesDICE is highly competitive to existing state-of-the-art approaches in confidence interval estimation. More importantly, we show how the belief distribution estimated by BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric, and we empirically demonstrate that this selection procedure significantly outperforms existing approaches, such as ranking policies according to mean or high-confidence lower bound value estimates.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

A Minimalist Approach to Offline Reinforcement Learning (2021, background): “This problem may be additionally amplified in the context of offline RL, where evaluating changes to implementation and hyperparameters is counter-intuitive to the nature of offline RL, which explicitly aims to eliminate environment interactions [Paine et al., 2020, Yang et al., 2020].”
Universal Off-Policy Evaluation (2021, background): “Instead of frequentist intervals, a Bayesian posterior distribution over the mean return and various statistics of that distribution can also be obtained [105].”