Confounding-Robust Policy Improvement

By Nathan Kallus, Angela Zhou. 2018

174 citations recorded by the source corpus.

We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding. Previous approaches, which assume unconfoundedness, i.e., that no unobserved confounders affect both the treatment assignment as well as outcome, can lead to policies that introduce harm rather than benefit when some unobserved confounding is present, as is generally the case with observational data. Instead, since policy value and regret may not be point-identifiable, we study a method that minimizes the worst-case estimated regret of a candidate policy against a baseline policy over an uncertainty set for propensity weights that controls the extent of unobserved confounding. We prove generalization guarantees that ensure our policy will be safe when applied in practice and will in fact obtain the best-possible uniform control on the range of all possible population regrets that agree with the possible extent of confounding. We develop efficient algorithmic solutions to compute this confounding-robust policy. Finally, we assess and compare our methods on synthetic and semi-synthetic data. In particular, we consider a case study on personalizing hormone replacement therapy based on observational data, where we validate our results on a randomized experiment. We demonstrate that hidden confounding can hinder existing policy learning approaches and lead to unwarranted harm, while our robust approach guarantees safety and focuses on well-evidenced improvement, a necessity for making personalized treatment policies learned from observational data reliable in practice.

Subfields: Economics and Econometrics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

Learning When-to-Treat Policies (2019, methodology): “The promise of personalized data-driven decision-making has led to a surge in interest in methods that leverage observational data to help inform how and to whom interventions should be applied (Athey and Wager, 2017; Bertsimas and Kallus, 2020; Dud́ık, Erhan, Langford, and Li, 2014; Elmachtoub and ”
Efficient Policy Learning (2017, background): “The main result of Kitagawa and Tetenov (2018) is that, if treatment propensities e(Xi) are known, then a variant of inverse-propensity weighted policy learning achieves regret on the order of √ VC(Π)/n.”
Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancing (2021, methodology): “Related work by [27, 28, 26, 34] takes the idea substantially further.”
Policy Evaluation with Latent Confounders via Optimal Balance (2019, background): “Additionally, there is recent work on robustness to unidentifiability due to unobserved confounding [20, 23, 24]; in contrast we focus on a setting where, while confounders are unobserved, effects are identifiable via proxies.”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “…al., 2019; Zhang et al., 2020c; Kumor et al., 2021), batch learning (Bannon et al., 2020), and partial observability settings (Forney et al., 2017; Kallus & Zhou, 2018; Zhang et al., 2019a; Lu et al., 2018) in the online environment (Lyle et al., 2021; Zhang & Bareinboim, 2018), (Wang et al.,…”
Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification (2021, unknown): “The tool of causality analysis has been successfully used in several areas, including explainable machine learning [40], fairness [24], natural language processing [61], reinforcement learning [20] and adversarial learning [22].”
Policy Learning With Observational Data (2017, background): “Kallus and Zhou (2020) consider the problem of policy learning under an approximate selection-on-observables assumption in the sense of Rosenbaum (2002).”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “…our setting is completely oﬀ-policy with no opportunity of collecting new data from arbitrary policies, as considered in, e.g. , Athey and Wager (2017); Kallus (2018); Kallus and Zhou (2018); Swaminathan and Joachims (2015) for H = 0 and Chen et al. (2019); Fujimoto et al. (2019) for H ≥ 1.”
Counterfactual risk assessments, evaluation, and fairness (2019, background): “[18] propose policy learning via minimax regret learning over uncertainty sets.”
Bayesian Robust Optimization for Imitation Learning (2020, background): “In reinforcement learning, it has been shown that optimizing the regret is more appropriate if a good baseline policy is available [29, 31, 40].”
A Survey on Causal Inference (2020, background): “However, when there exist unobserved confounders, it can lead to policies that introduce harm rather than benefit, as is generally the case with observational data [65].”