Learning When-to-Treat Policies

By Xinkun Nie, Emma Brunskill, Stefan Wager. 2019

100 citations recorded by the source corpus.

Abstract Many applied decision-making problems have a dynamic component: The policymaker needs not only to choose whom to treat, but also when to start which treatment. For example, a medical doctor may choose between postponing treatment (watchful waiting) and prescribing one of several available treatments during the many visits from a patient. We develop an “advantage doubly robust” estimator for learning such dynamic treatment rules using observational data under the assumption of sequential ignorability. We prove welfare regret bounds that generalize results for doubly robust learning in the single-step setting, and show promising empirical performance in several different contexts. Our approach is practical for policy optimization, and does not need any structural (e.g., Markovian) assumptions. Supplementary materials for this article are available online.

Subfields: Economics and Econometrics, Statistics and Probability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

In Causal Inference & Discovery

Citations in the corpus

Benchmarks for Deep Off-Policy Evaluation (2021, background): “…Brunskill, 2016; Swaminathan & Joachims, 2015; Jiang & Li, 2015; Wang et al., 2017; Yang et al., 2020) and practical progress (Gilotte et al., 2018; Nie et al., 2019; Kalashnikov et al., 2018) on OPE algorithms has been made in a range of different domains, there are few broadly accepted…”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations [31, 57, 3, 1], and the increasing importance of off-policy learning and counterfactual reasoning [12, 52, 37, 30, 34, 41].”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “This paradigm is useful in a diverse array of application domains such as healthcare [51, 15, 35], autonomous driving [57, 4, 36], and recommendation systems [47, 12, 49].”
Policy Learning With Observational Data (2017, background): “Nie, Brunskill, and Wager (2019) proposed a method for learning observational stopping rules from observational data that is both computationally feasible and robust to confounding.”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “Nie et al. (2019) do extend these types of regret results to an RL problem but where the problem has a special when-to-treat structure, not the general MDP case.”