Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization

By Nan Jiang, Jiawei Huang. 2020

17 citations recorded by the source corpus.

We study minimax methods for off-policy evaluation (OPE) using value-functions and marginalized importance weights. Despite that they hold promises of overcoming the exponential variance in traditional importance sampling, several key problems remain: (1) They require function approximation and are generally biased. For the sake of trustworthy OPE, is there any way to quantify the biases? (2) They are split into two styles ("weight-learning" vs "value-learning"). Can we unify them? In this paper we answer both questions positively. By slightly altering the derivation of previous methods (one from each style; Uehara et al., 2019), we unify them into a single confidence interval(CI) that comes with a special type of double robustness: when either the value-function or importance weight class is well-specified, the CI is valid and its length quantifies the misspecification of the other class. Our CI also provides a unified view of and new insights to some recent methods, and we further explore the implications of our results on exploration and exploitation in off-policy policy optimization with insufficient data coverage.

Subfields: Environmental Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Environmental Engineering

In Statistics and Probability

Citations in the corpus

Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, background): “14See the parallel work by Nachum et al. (2019) and the followup work of Jiang and Huang (2020) for some intriguing discussions on this matter.”
Universal Off-Policy Evaluation (2021, background): “(B) Comparison of UnO with prior work: Recent works for bounding the mean [45, 35] assume no confounding and Markovian structure.”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, unknown): “…Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara and Jiang, 2019, Kallus and Uehara, 2020, Jiang and Huang, 2020, Feng et al., 2020, Yang et al., 2020, Nachum et al., 2019a, Zhang et al., 2020a,b].”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, methodology): “Bypassing such exponential dependency requires non-trivial function approximation assumptions [Jiang and Huang, 2020, Feng et al., 2020, Liu et al., 2018].”