New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes

By Ying‐Qi Zhao, Donglin Zeng, Eric B. Laber, Michael R. Kosorok. 2015

263 citations recorded by the source corpus.

Dynamic treatment regimes (DTRs) are sequential decision rules for individual patients that can adapt over time to an evolving illness. The goal is to accommodate heterogeneity among patients and find the DTR which will produce the best long-term outcome if implemented. We introduce two new statistical learning methods for estimating the optimal DTR, termed backward outcome weighted learning (BOWL), and simultaneous outcome weighted learning (SOWL). These approaches convert individualized treatment selection into an either sequential or simultaneous classification problem, and can thus be applied by modifying existing machine learning techniques. The proposed methods are based on directly maximizing over all DTRs a nonparametric estimator of the expected long-term outcome; this is fundamentally different than regression-based methods, for example, Q-learning, which indirectly attempt such maximization and rely heavily on the correctness of postulated regression models. We prove that the resulting rules are consistent, and provide finite sample bounds for the errors using the estimated rules. Simulation results suggest the proposed methods produce superior DTRs compared with Q-learning especially in small samples. We illustrate the methods using data from a clinical trial for smoking cessation. Supplementary materials for this article are available online.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Data (2017, methodology): “In each step, a parametric prediction model is fitted for the value function (Q-learning [15]); or a classification model is constructed for modeling the decision policies directly (Outcome Weighted Learning (OWL) [16]).”
Learning When-to-Treat Policies (2019, methodology): “Zhao et al. (2015) provided regret bounds for nonparametric IPW-type estimators in a dynamic setting.”
A Theoretical Analysis of Deep Q-Learning (2019, background): “d Murphy (2011); Zhao et al. (2011); Zhang et al. (2012); Zhao et al. (2012); Goldberg and Kosorok (2012); Nahum-Shani et al. (2012); Goldberg et al. (2013); Schulte et al. (2014); Song et al. (2015);Zhao et al. (2015); Linn et al. (2017);Zhou et al. (2017); Shi et al. (2018); Zhu et al. (2019) and ”
Interactive Q-learning for Quantiles (2017, methodology): “Optimization of threshold-exceedance probabilities can be framed as a special case optimizing the mean of a binary outcome, for which several methods exist, including the classification-based outcome weighted learning (Zhang et al. 2012b; Zhao et al. 2015).”
Robust Hybrid Learning for Estimating Personalized Dynamic Treatment Regimens (2016, methodology): “Thirdly, and most importantly, for multiple-stage trials, AMOL estimates optimal DTRs via a backward induction procedure that makes non-trivial extension of Zhao et al. (2014) to boost eﬃciency through augmentation and integration with Q-learning.”
Dynamic treatment regimes: technical challenges and applications (2014, background): “Recent statistical work in this area includes marginal structural mean models (Robins et al., 2008; Orellana et al., 2010), augmented value maximization (Zhang et al., 2012, 2013), and outcome weighted learning (Zhao et al., 2012, 2013).”
Precision Medicine. (2019, methodology): “For these reasons, direct search estimators based on Vn(d) work best for settings where T is small, e.g., T = 2; however, in such settings OWL and related methods can offer significant gains in terms of robustness and marginal mean outcome (see Zhang et al 2013; Zhao et al 2015a).”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “missclassication rate and applies either discrete-optimization or classication algorithms to optimize a plugin estimator of the marginal mean outcome (Zhao et al., 2012; Zhang et al., 2012b,a, 2013; Zhao et al., 2015). Regression-based or indirect estimators comprise a second class of estimators of ”
Estimation of Optimal Dynamic Treatment Regimes (2014, methodology): “Methods for estimating V(d), include inverse probability-weighting [9,14]; augmented inverse probability-weighting [10,13]; and marginal structural mean models [7].”
Residual Weighted Learning for Estimating Individualized Treatment Rules (2015, background): “Zhao et al. (2014) has extended OWL to dynamic treatment regimes.”
Learning Optimal Individualized Treatment Rules from Electronic Health Record Data (2016, methodology): “Statistical methods for estimating optimal medical treatment and disease management strategy according to patient-speciﬁc characteristics for each individual have received extensive attention in the recent literature [3], [4], [5], [6].”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “…2012; Moodie et al., 2012, 2014; Laber et al., 2014; Tian et al., 2014) or policy-search-based (Orellana et al., 2010; Zhang et al., 2012, 2013; Zhao et al., 2012, 2014, 2015; Laber and Zhao, 2015), though this division 18 © 2017, The International Biometric Society is somewhat superficial…”
Doubly Robust Learning for Estimating Individualized Treatment with Censored Data (2014, unknown): “Classificationbased approaches have been shown to work well in settings without censoring (Zhao et al., 2012; Zhang et al., 2012a; Kang et al., 2014; Zhao et al., 2014).”
Interpretable Dynamic Treatment Regimes (2016, methodology): “The first four came from Zhao et al. (2015) and the fifth was adapted from Murphy (2003).”
Augmented outcome-weighted learning for estimating optimal dynamic treatment regimens (2018, methodology): “However, in the aforementioned work, 18 since the weights at each stage of the estimation must be the optimal outcome increment in the future stages, only patients whose later treatments are optimal can be used for estimation.”