Off-Policy Evaluation via the Regularized Lagrangian

By Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, Dale Schuurmans. 2020

122 citations recorded by the source corpus.

The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data. While these estimators all perform some form of stationary distribution correction, they arise from different derivations and objective functions. In this paper, we unify these estimators as regularized Lagrangians of the same linear program. The unification allows us to expand the space of DICE estimators to new alternatives that demonstrate improved performance. More importantly, by analyzing the expanded space of estimators both mathematically and empirically we find that dual solutions offer greater flexibility in navigating the tradeoff between optimization stability and estimation bias, and generally provide superior estimates in practice.

Subfields: Software, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Software

1. Ofir Nachum
2. Nan Jiang
3. Sergey Levine
4. Aviral Kumar
5. George Tucker
6. Bo Dai
7. Hoang Minh Le
8. Yisong Yue

In Statistics and Probability

1. George Tucker
2. Nan Jiang
3. Ofir Nachum
4. Sergey Levine
5. Yisong Yue
6. Hoang Minh Le
7. Cameron Voloshin
8. Bo Dai

Citations in the corpus

• Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “We use the implementation from Yang et al. (2020) corresponding to the algorithm BestDICE.4 Variational Power Method (VPM) This method runs a variational power iteration algorithm to estimate the importance weights dπ(s, a)/dπB (s, a) without the knowledge of the behavior policy.”
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “Starting with IH, this style of minimax estimator has recently attracted significant attention in OPE literature, including state-action extension of IH [54, 25] and DICE family of estimators [40, 61, 59, 10].”
• Instabilities of Offline RL with Pre-Trained Neural Representation (2021, unknown): “…Guo et al., 2017, Wang et al., 2017, Liu et al., 2018, Farajtabar et al., 2018, Xie et al., 2019, Kallus and Uehara, 2019, Liu et al., 2019, Uehara and Jiang, 2019, Kallus and Uehara, 2020, Jiang and Huang, 2020, Feng et al., 2020, Yang et al., 2020, Nachum et al., 2019a, Zhang et al., 2020a,b].”
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization (2021, unknown): “, 2014) and Lagrangian duality (Nachum et al., 2019; Yang et al., 2020; Uehara and Jiang, 2019).”
• Offline Policy Selection under Uncertainty (2020, methodology): “In addition to a plethora of techniques for providing point estimates of this groundtruth value (Dudík et al., 2011; Bottou et al., 2013; Jiang and Li, 2015; Thomas and Brunskill, 2016; Kallus and Uehara, 2020; Nachum et al., 2019; Zhang et al., 2020; Yang et al., 2020), there is also a growing body”
• Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions (2022, methodology): “Regularization in MIS The use of regularization is very common in the MIS literature, especially in DICE algorithms (Nachum et al., 2019a;b; Yang et al., 2020).”
• Is Pessimism Provably Efficient for Offline RL? (2020, background): “…et al., 2018; Xie et al., 2019; Nachum et al., 2019a;b; Tang et al., 2019; Kallus and Uehara, 2019; 2020; Jiang and Huang, 2020; Uehara et al., 2020; Duan et al., 2020; Yin and Wang, 2020; Yin et al., 2020; Nachum and Dai, 2020; Yang et al., 2020a; Zhang et al., 2020b) and the references therein.”