Policy Evaluation and Optimization with Continuous Treatments

By Nathan Kallus, Angela Zhou. 2018

157 citations recorded by the source corpus.

We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments. Previous work for discrete treatment/action spaces focuses on inverse probability weighting (IPW) and doubly robust (DR) methods that use a rejection sampling approach for evaluation and the equivalent weighted classification problem for learning. In the continuous setting, this reduction fails as we would almost surely reject all observations. To tackle the case of continuous treatments, we extend the IPW and DR approaches to the continuous setting using a kernel function that leverages treatment proximity to attenuate discrete rejection. Our policy estimator is consistent and we characterize the optimal bandwidth. The resulting continuous policy optimizer (CPO) approach using our estimator achieves convergent regret and approaches the best-in-class policy for learnable policy classes. We demonstrate that the estimator performs well and, in particular, outperforms a discretization-based benchmark. We further study the performance of our policy optimizer in a case study on personalized dosing based on a dataset of Warfarin patients, their covariates, and final therapeutic doses. Our learned policy outperforms benchmarks and nears the oracle-best linear policy.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Causal Inference & Discovery

Citations in the corpus

Balanced Linear Contextual Bandits (2018, neutral): “…(Li et al. 2012), (Dud ´ ık et al. 2014), (Li et al. 2014), (Swaminathan and Joachims 2015), (Jiang and Li 2016), (Thomas and Brunskill 2016), (Athey and Wager 2017), (Kallus 2017), (Wang, Agarwal, and Dud ´ ık 2017), (Deshpande et al. 2017), (Kallus and Zhou 2018), (Zhou, Athey, and Wager 2018).”
Double/De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers (2018, background): “Prominent examples of the debiased CTE literature include Rubin and van der Laan (2006), Dı́az and van der Laan (2013), Galvao and Wang (2015), Kennedy et al. (2017), Kallus and Zhou (2018), and Colangelo and Lee (2020).”
Estimation Considerations in Contextual Bandits (2017, background): “There is also a significant body of research that leverages balancing for offline evaluation and learning of contextual bandit or reinforcement learning policies from logged data [57, 28, 44, 27, 43, 59, 37, 60, 10, 38, 64, 26, 39, 58, 66].”
Balanced Policy Evaluation and Learning (2017, background): “Kallus and Zhou [26] generalize the IPW approach to a continuum of treatments.”
Confounding-Robust Policy Improvement (2018, methodology): “A variety of methods have been proposed to address the corresponding problem of policy learning from observational data (Beygelzimer and Langford 2009, Dudik et al. 2014, Kallus 2017a,b, Kallus and Zhou 2018, Kitagawa and Tetenov 2015, Wager and Athey 2017a).”
Policy Evaluation with Latent Confounders via Optimal Balance (2019, methodology): “In particular, this enables the use of inverse propensity score (IPS) estimators of policy value [4, 21, 29] that eschew the need to actually fit outcome prediction models and doubly robust estimators that work even if such models are misspecified [8].”