Reinforcement learning design for cancer clinical trials

By Yufan Zhao. 2009

283 citations recorded by the source corpus.

We develop reinforcement learning trials for discovering individualized treatment regimens for life‐threatening diseases such as cancer. A temporal‐difference learning method called Q‐learning is utilized that involves learning an optimal policy from a single training set of finite longitudinal patient trajectories. Approximating the Q‐function with time‐indexed parameters can be achieved by using support vector regression or extremely randomized trees. Within this framework, we demonstrate that the procedure can extract optimal strategies directly from clinical data without relying on the identification of any accurate mathematical models, unlike approaches based on adaptive design. We show that reinforcement learning has tremendous potential in clinical research because it can select actions that improve outcomes by taking into account delayed effects even when the relationship between actions and outcomes is not fully known. To support our claims, the methodology's practical utility is illustrated in a simulation analysis. In the immediate future, we will apply this general strategy to studying and identifying new treatments for advanced metastatic stage IIIB/IV non‐small cell lung cancer, which usually includes multiple lines of chemotherapy treatment. Moreover, there is significant potential of the proposed methodology for developing personalized treatment strategies in other cancers, in cystic fibrosis, and in other life‐threatening diseases. Copyright © 2009 John Wiley & Sons, Ltd.

Subfields: Cancer Research, Molecular Biology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Molecular Biology

In Statistics and Probability

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “nsidering censoring problem in multiple lines of treatment in advanced NSCLC; using overall survival time as the net reward. F¨urnkranz et al. [52], Cheng et al. [89] PI N/A PRL ODE model proposed in [83] Combining preference learning and RL for optimal therapy design in cancer treatment, but only i”
Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Data (2017, background): “Methods to identify the optimal DTRs have recently received attentions in the statistical community [7], [8], [9], [10], [11], [12], [13], [14], and most of exsiting methods focused on data from randomized clinical trials.”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, background): “There has been a surge of interest in recent years within the medical community regarding the application of Reinforcement Learning (RL) techniques to optimize treatment policies (Thall and Wathan 2000; Ernst et al. 2006; Murphy et al. 2007; Guez et al. 2008; Zhao et al. 2009).”
APRIL: Active Preference-learning based Reinforcement Learning (2012, background): “rticial problem. Its integration within April is thereafter studied and a proof of concept of April is given on the classical mountain car problem, and the cancer treatment testbed rst introduced by [32]. This paper is organized as follows. Section 2 brie y presents Ppl for selfcontainedness and dis”
Using Decision Lists to Construct Interpretable and Parsimonious Treatment Regimes (2015, background): “e modeled portions of the 2 conditional distribution of the outcome. Thus, a goal of many regression-based estimators is to ensure correct model specication under a large class of generative models (Zhao et al., 2009; Qian and Murphy, 2011; Moodie et al., 2013; Laber et al., 2014; Taylor et al., 201”
Reinforcement Learning Strategies for Clinical Trials in Non-small Cell Lung Cancer (2011, methodology): “Our experience so far indicates that both SVR and ERT work quite well and their accuracy is approximately equivalent, although ERT is more computationally intense (Zhao et al., 2009).”
Interactive machine learning for health informatics: when do we need the human-in-the-loop? (2016, methodology): “In an interesting experiment, they applied a model developed by [68] to a simulation of optimal therapy design in cancer treatment.”
Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm (2014, methodology): “A corresponding simulation model based on first-order difference equations was originally introduced in [42].”
Constructing Dynamic Treatment Regimes in Infinite-Horizon Settings (2014, methodology): “ (2003) uses a least square characterization method (Moodie et al., 2007). Q-learning, a reinforcement learning algorithm, is also widely used in constructing the optimal regime (Murphy et al., 2006; ZHAO et al., 2009; Chakraborty et al., 2010; NahumShani et al., 2012). Q-learning is an extension of”
Estimation of Optimal Dynamic Treatment Regimes (2014, background): “The Q-learning algorithm requires postulated models for the Q-functions; we consider linear working models of the form Qj(Hj, Aj; uj) =H T j aj +AjH T j bj, where uj = (a T j , b T j ) T, and Hj1, Hj2 are known features constructed from Hj. Q-learning algorithm mimics the dynamic programming solutio”
A Reinforcement Learning Design for HIV Clinical Trials (2014, methodology): “Like Zhao et al. (2009), we use a composite function to define the reward the agent receives at each time step during learning.”
Estimating Optimal Treatment Regimes from a Classification Perspective (2012, methodology): “The method of Zhao et al. (2012) estimates the optimal treatment regime by maximizing a concave relaxation of the above IPWE estimator.”
Adaptive treatment of anemia on hemodialysis patients: A reinforcement learning approach (2011, unknown): “[12], in all cases with promising results.”
Policy Search in a Space of Simple Closed-form Formulas: Towards Interpretability of Reinforcement Learning (2012, background): “This is especially the case in the context of medical applications involving patients’ health [22, 13, 30].”
Doubly Robust Learning for Estimating Individualized Treatment with Censored Data (2014, unknown): “2 in Zhao et al. (2012) and are thus omitted.”
Preference-Based Policy Iteration: Leveraging Preference Learning for Reinforcement Learning (2011, methodology): “Again following [17], we let c0 = −4, c1 = c2 = 1.”