Penalized Q-Learning for Dynamic Treatment Regimens

By Rui Song, W. Wang, Donglin Zeng, Michael R. Kosorok. 2015

101 citations recorded by the source corpus.

A dynamic treatment regimen incorporates both accrued information and long-term effects of treatment from specially designed clinical trials. As these trials become more and more popular in conjunction with longitudinal data from clinical studies, the development of statistical inference for optimal dynamic treatment regimens is a high priority. In this paper, we propose a new machine learning framework called penalized Q-learning, under which valid statistical inference is established. We also propose a new statistical procedure: individual selection and corresponding methods for incorporating individual selection within penalized Q-learning. Extensive numerical studies are presented which compare the proposed methods with existing methods, under a variety of scenarios, and demonstrate that the proposed approach is both inferentially and computationally superior. It is illustrated with a depression clinical trial study.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “tic empirical studies of Q-learning and Advantage-learning (A-learning) [172] methods and illustrated their performance using data from an MDD study. Other approaches include the penalized Q-learning [173], the Augmented Multistage Outcome-Weighted Learning (AMOL) [174], the budgeted learning algori”
Interactive Q-learning for Quantiles (2017, background): “This leads to nonregular estimators for parameters that index the optimal regime and complicates the search for models that fit the data well since many standard regression modeling diagnostics are invalid (Robins 2004; Chakraborty, Murphy, and Strecher 2010; Laber et al. 2014c; Song et al. 2015).”
Set-valued dynamic treatment regimes for competing outcomes (2012, background): “, 2012), depression (Schulte et al., 2012; Song et al., 2012), HIV infection (Moodie, Richardson, and Stephens, 2007), schizophrenia (Shortreed et al.”
Interactive Q-learning for Probabilities and Quantiles (2014, methodology): “…Q-learning (Watkins, 1989; Watkins and Dayan, 1992; Murphy, 2005a), regularized Q-learning (Moodie and Richardson, 2010; Chakraborty et al., 2010; Song et al., 2011; Goldberg et al., 2013), Interactive Q-learning (Laber et al., 2014), g-estimation (Robins, 2004), A-learning (Murphy, 2003),…”
Q-Learning: Theory and Applications (2020, background): “Table 2 serves as a reference for the correspondence between the respective terminologies 241 .”
Sparse concordance-assisted learning for optimal treatment decision (2018, methodology): “Song et al. (2015) extended this method to penalized outcome weighted learning (POWL).”