Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models

By Michael Oberst, David A. Sontag. 2019

200 citations recorded by the source corpus.

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual trajectories in finite partially observable Markov Decision Processes (POMDPs). We see this as a useful procedure for off-policy "debugging" in high-risk settings (e.g., healthcare); by decomposing the expected difference in reward between the RL and observed policy into specific episodes, we can identify episodes where the counterfactual difference in reward is most dramatic. This in turn can be used to facilitate review of specific episodes by domain experts. We demonstrate the utility of this procedure with a synthetic environment of sepsis management.

Subfields: Economics and Econometrics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, background): “Later, [23] discusses the case with categorical variables.”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, methodology): “Tackling this challenge, possibly using techniques from causal inference [50, 42], is an important future direction.”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, methodology): “This differs from Oberst and Sontag (2019) who focus solely on a new approach to generating counterfactual trajectories under a proposed policy.”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, methodology): “This has seen counterfactuals applied to medicine (Constantinou et al., 2016; Schulam and Saria, 2017; Richens et al., 2019; Oberst and Sontag, 2019), advertisement and search (Bottou et al.”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, background): “Oberst & Sontag (2019) proposed framing the OPE problem as a structural causal model, which enabled them to identify trajectories where the predicted counterfactual trajectories under an evaluation policy differs substantially from the observed data collected under the behavior policy.”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, background): “Oberst & Sontag (2019) also investigate counterfactuals in reinforcement learning, point out the issue of non-identifiability of the correct SCM, and suggest a sufficient condition for identifiability; we discuss this issue in appendix G. Closely related to our work is Hindsight Credit Assignment,…”
Deep Structural Causal Models for Tractable Counterfactual Inference (2020, background): “In particular, discrete mechanisms with a Gumbel–max parametrisation have been shown to lead to counterfactuals satisfying desirable properties [57].”
Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “The machine learning community has also made efforts in treatment effect estimation [39, 40, 41] and real-world applications [42, 43, 44, 28, 45, 46, 47].”