Representation Balancing MDPs for Off-Policy Policy Evaluation

By Yao Liu, Omer Gottesman, Aniruddh Raghu, Matthieu Komorowski, Aldo A. Faisal, Finale Doshi-Velez, Emma Brunskill. 2018

75 citations recorded by the source corpus.

We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value estimates from MDP models. Using this upper bound as an objective, we develop a learning algorithm of an MDP model with a balanced representation, and show that our approach can yield substantially lower MSE in common synthetic benchmarks and a HIV treatment simulation domain.

Subfields: Artificial Intelligence, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Unlike many previous approaches to off-policy policy evaluation (Peshkin & Shelton, 2002; Thomas et al., 2015; Liu et al., 2018), our work focuses on constraining the policy to a subset of policies which can be adequately evaluated, rather than the process of evaluation itself.”
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, methodology): “Model-based methods for off-policy evaluation have recently been improved to account for the distribution shift between the data-collecting policy and the policy to be evaluated (Johansson et al., 2016; Liu et al., 2018).”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “…factor introduced by ∑∞ t=0 γ t. Similar to Bellman equation for value function, the state density function can also be viewed as a fixed point to the following recursive equation (Liu et al., 2018a, Lemma 3): dπ(s ′) = (1− γ)µ0(s′) + γT πdπ(s′), where T πdπ(s′) := ∑ s,a T (s′|s, a)π(a|s)dπ(s).”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “…policy, for a sequential decision-making problem using data generated by the log of another policy, known as the behavior policy (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016; Xie et al., 2019).”
Learning When-to-Treat Policies (2019, methodology): “In observational studies treatment probabilities need to be estimated from data, and it is known that the variant of (6) with estimated weights γ̂ (i)t (π) can perform poorly with even mild estimation error (see, e.g., Y. Liu et al. 2018).”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, methodology): “The main issue of this approach, when applying to the episodic MDP with large action space is that the variance of the importance weights grows exponentially in H [Liu et al., 2018a], which makes the sample complexity exponentially worse than the model-based approaches, when they are applicable.”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “Prior works [7, 10, 13, 21, 28, 34] do so with importance sampling (IS) [11], MDP modeling, or both.”
Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes (2019, background): “…policy, for a sequential decision-making problem using data generated by the log of another policy, known as the behavior policy (Jiang and Li, 2016; Li et al., 2015; Liu et al., 2018b; Mahmood et al., 2014; Munos et al., 2016; Precup et al., 2000; Thomas and Brunskill, 2016; Xie et al., 2019).”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “Evaluation in oﬀ-policy settings often uses a statistical technique known as oﬀ-policy policy evaluation (example algorithms include [178, 133]).”
Model-Based Reinforcement Learning for Sepsis Treatment (2018, methodology): “We do not use recent model-based estimators, introduced by [11], because of the challenge in learning full environment models in this domain.”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “Several other works (Guo et al., 2017; Hallak & Mannor, 2017; Liu et al., 2018a; Gelada & Bellemare, 2019; Nachum et al., 2019) have been proposed to address the high variance issue in the long-horizon problems.”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “One of the notable prior works is Representation Balancing MDP (RepBM) (Liu et al., 2018b), which regularizes the representation learning of the model to be invariant between the distributions.”