Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning

By Nathan Kallus, Masatoshi Uehara. 2019

61 citations recorded by the source corpus.

Off-policy evaluation (OPE) in both contextual bandits and reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. The problem's importance has attracted many proposed solutions, including importance sampling (IS), self-normalized IS (SNIS), and doubly robust (DR) estimates. DR and its variants ensure semiparametric local efficiency if Q-functions are well-specified, but if they are not they can be worse than both IS and SNIS. It also does not enjoy SNIS's inherent stability and boundedness. We propose new estimators for OPE based on empirical likelihood that are always more efficient than IS, SNIS, and DR and satisfy the same stability and boundedness properties as SNIS. On the way, we categorize various properties and classify existing estimators by them. Besides the theoretical guarantees, empirical studies suggest the new estimators provide advantages.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

1. Sergey Levine
2. Aviral Kumar
3. George Tucker
4. Justin Fu
5. Ofir Nachum
6. Shixiang Shane Gu
7. Chelsea Finn
8. Pieter Abbeel

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Recent work (Mousavi et al., 2020; Kallus and Uehara, 2019a,b; Tang et al., 2019; Uehara and Jiang, 2019) has refined this approach, in particular removing the need to have access to πβ .”
• Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “Before turning to developing an efficient estimator under the MDP model, we briefly review how we can extend the efficient finite-horizon DRL estimators of Kallus and Uehara (2019a) to be efficient in the infinite-horizon NMDP and TMDP settings.”
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, background): “While much work has studied efficient estimation underM1 (Dudik et al., 2014; Jiang and Li, 2016; Kallus and Uehara, 2019; Thomas and Brunskill, 2016), work on M2 has been restricted to the parametric, finite-state-finite-action case (Jiang and Li, 2016) and no globally efficient estimators have bee”
• Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes (2019, methodology): “We show that adaptations of the doubly robust (DR) estimator in NMDPs (Jiang and Li, 2016) and in MDPs (Kallus and Uehara, 2019b) to the infinite horizon case achieve these bounds, i.e. , are efficient in the near-on-policy setting.”
• Policy Evaluation with Latent Confounders via Optimal Balance (2019, background): “The doubly robust estimator is known to be consistent when either the weighted or direct estimator is consistent and can attain local efficiency [19, 39].”
• A Survey on Causal Inference (2020, methodology): “Later, various methods [10, 19, 74, 88, 154, 155, 158, 179] are proposed to improve those two methods with different settings.”
• Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “(20) Here, the terms from (18) to (19) are the same as the decomposition in Kallus and Uehara (2019a).”