Q-LEARNING WITH CENSORED DATA

By Yair Goldberg, Michael R. Kosorok. 2012

149 citations recorded by the source corpus.

We develop methodology for a multistage-decision problem with flexible number of stages in which the rewards are survival times that are subject to censoring. We present a novel Q-learning algorithm that is adjusted for censored data and allows a flexible number of stages. We provide finite sample bounds on the generalization error of the policy learned by the algorithm, and show that when the optimal Q-function belongs to the approximation space, the expected survival time for policies obtained by the algorithm converges to that of the optimal policy. We simulate a multistage clinical trial with flexible number of stages and apply the proposed censored-Q-learning algorithm to find individualized treatment regimens. The methodology presented in this paper has implications in the design of personalized medicine trials in cancer and in other life-threatening diseases.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “ribution to ﬁll in missing values, the censoring data problem is far more challenging, calling for more sophisticated techniques for state representation and value estimation in such ﬂexible settings [96], [97]. Most existing work deﬁnes the states over the processed medical data with raw physiologi”
New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes (2015, methodology): “Methods have been developed for the multidecision problem within the Q-learning framework, where survival times are the outcome of interest (Zhao et al. 2011; Goldberg and Kosorok 2012).”
Doubly Robust Learning for Estimating Individualized Treatment with Censored Data (2014, unknown): “Thus, there is increasing interest in dynamic treatment regimens (Thall et al., 2002; Murphy, 2003; Robins, 2004; Moodie et al., 2007; Zhao et al., 2011; Goldberg and Kosorok, 2012; Huang et al., 2014).”
Estimation of Optimal Dynamic Treatment Regimes (2014, methodology): “Other advantages of Q-learning are as follows: (1) it can be extended to accommodate discrete outcomes [45,50], censored outcomes [49], and competing outcomes [54,55]; (2) measures of goodness of fit and visual diagnostics can be used to assess the quality of the fitted regression models in each sta”
Constructing Dynamic Treatment Regimes in Infinite-Horizon Settings (2014, unknown): “al., 2009; Chakraborty et al., 2010; NahumShani et al., 2012). Q-learning is an extension of the standard regression method that can be used with longitudinal data in which treatments vary over time. Goldberg and Kosorok (2011) introduce a new Q-learning method that can be used when individuals are ”
Precision Medicine. (2019, methodology): “…Q-learning relies on a series of regression models, it is easily extensible to a variety of models and data structures (Zhao et al 2009; Goldberg and Kosorok 2012; Moodie et al 2014) and allows the user to interactively construct and critique the models used for the Qfunctions (Rich et…”
Deep Reinforcement Learning: An Overview (2017, unknown): “Goldberg and Kosorok (2012)”
Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules (2017, background): “Estimators are typically broadly categorized as regression-based (Murphy, 2005; Henderson et al., 2010; Qian and Murphy, 2011; Zhao et al., 2011; Goldberg and Kosorok, 2012; Moodie et al., 2012, 2014; Laber et al., 2014; Tian et al., 2014) or policy-search-based (Orellana et al., 2010; Zhang et…”
Interactive Q-learning for Quantiles (2017, background): “Direct-search and regression-based estimators have been extended to handle survival outcomes (Goldberg and Kosorok 2012; Huang and Ning 2012; Huang, Ning, and Wahed 2014), high-dimensional data (McKeague and Qian 2014), missing data (Shortreed et al. 2014), multiple outcomes (Laber, Lizotte, and…”
A Theoretical Analysis of Deep Q-Learning (2019, background): “nesses. There is a huge body of literature on this line of research. See, e.g., Murphy (2003); Zhao et al. (2009); Qian and Murphy (2011); Zhao et al. (2011); Zhang et al. (2012); Zhao et al. (2012); Goldberg and Kosorok (2012); Nahum-Shani et al. (2012); Goldberg et al. (2013); Schulte et al. (2014”
Interactive Q-learning for Probabilities and Quantiles (2014, background): “Extensions exist for estimating dynamic treatment regimes under censoring (Goldberg and Kosorok, 2012), high-dimensional data (McKeague and Qian, 2013), and missing data (Shortreed et al.”