Robust Hybrid Learning for Estimating Personalized Dynamic Treatment Regimens
By Ying Liu, Yuanjia Wang, Michael R. Kosorok, Ying‐Qi Zhao, Donglin Zeng. 2016
34 citations recorded by the source corpus.
Dynamic treatment regimens (DTRs) are sequential decision rules tailored at each stage by potentially time-varying patient features and intermediate outcomes observed in previous stages. The complexity, patient heterogeneity and chronicity of many diseases and disorders call for learning optimal DTRs which best dynamically tailor treatment to each individual's response over time. Proliferation of personalized data (e.g., genetic and imaging data) provides opportunities for deep tailoring as well as new challenges for statistical methodology. In this work, we propose a robust hybrid approach referred as Augmented Multistage Outcome-Weighted Learning (AMOL) to integrate outcome-weighted learning and Q-learning to identify optimal DTRs from the Sequential Multiple Assignment Randomization Trials (SMARTs). We generalize outcome weighted learning (O-learning; Zhao et al.~2012) to allow for negative outcomes; we propose methods to reduce variability of weights in O-learning to achieve numeric stability and higher efficiency; finally, for multiple-stage SMART studies, we introduce doubly robust augmentation to machine learning based O-learning to improve efficiency by drawing information from regression model-based Q-learning at each stage. The proposed AMOL remains valid even if the Q-learning model is misspecified. We establish the theoretical properties of AMOL, including the consistency of the estimated rules and the rates of convergence to the optimal value function. The comparative advantage of AMOL over existing methods is demonstrated in extensive simulation studies and applications to two SMART data sets: a two-stage trial for attention deficit and hyperactive disorder (ADHD) and the STAR*D trial for major depressive disorder (MDD).
Subfields: Control and Systems Engineering, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Control and Systems Engineering
In Statistics and Probability
Citations in the corpus
- Reinforcement Learning in Healthcare: A Survey (2019, methodology): “Other approaches include the penalized Q-learning [173], the Augmented Multistage Outcome-Weighted Learning (AMOL) [174], the budgeted learning algorithm [175], and the Censored Q-learning algorithm [97].”
- Learning Optimal Individualized Treatment Rules from Electronic Health Record Data (2016, methodology): “Even if a nonparametric learning algorithm is used, the method models the Q-function and its contrasts that are not explicitly related to the optimization of the objective function in (1) and this limitation was shown to result in suboptimal ITRs due to overfitting the regression model in [5], [6].”