Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

By Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, Qiang Liu. 2019

72 citations recorded by the source corpus.

Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to obtain higher accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or the value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Reinforcement Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Reinforcement Learning

In Causal Inference & Discovery

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Recent work (Mousavi et al., 2020; Kallus and Uehara, 2019a,b; Tang et al., 2019; Uehara and Jiang, 2019) has refined this approach, in particular removing the need to have access to πβ .”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, neutral): “Applying the Lagrangian to (82) will result in objectives similar to those that have appeared in Kallus and Uehara (2019b); Tang et al. (2019).”
Off-Policy Evaluation via the Regularized Lagrangian (2020, background): “Recently, a series of works were proposed to address these challenges [14, 17, 26].”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “…the reward and cost from the perspective of an optimization criterion, in which they consider the safety constraints via the duality property [162], [163], [164], [165] with regard to the state density functions, rather than the cost functions, reward functions and value functions [3], [52], [166].”
Representation Balancing Offline Model-based Reinforcement Learning (2021, unknown): “Recently, techniques of computing a stationary DIstribution Correction Estimation (DICE) have made remarkable progress that effectively addresses the curse of horizon (Liu et al., 2018a; Nachum et al., 2019a; Tang et al., 2020; Zhang et al., 2020; Mousavi et al., 2020).”
Is Pessimism Provably Efficient for Offline RL? (2020, background): “…Munos and Szepesv´ari, 2008; Farahmand et al., 2010; 2016; Scherrer et al., 2015; Liu et al., 2018; Nachum et al., 2019a;b; Chen and Jiang, 2019; Tang et al., 2019; Kallus and Uehara, 2019; 2020; Fan et al., 2020; Xie and Jiang, 2020a;b; Jiang and Huang, 2020; Uehara et al., 2020; Duan et al.,…”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “[222] propose the DCRL method that is to optimise the reward and cost from the perspective of an optimisation criterion based on optimisation theory, in which they consider the safety constraints via the duality property [65, 196, 197, 268] with regard to the state density functions, rather than the”