Reinforcement Learning Strategies for Clinical Trials in Non-small Cell Lung Cancer

By Yufan Zhao, Donglin Zeng, M. Socinski, Michael R. Kosorok. 2011

219 citations recorded by the source corpus.

Summary Typical regimens for advanced metastatic stage IIIB/IV nonsmall cell lung cancer (NSCLC) consist of multiple lines of treatment. We present an adaptive reinforcement learning approach to discover optimal individualized treatment regimens from a specially designed clinical trial (a “clinical reinforcement trial”) of an experimental treatment for patients with advanced NSCLC who have not been treated previously with systemic therapy. In addition to the complexity of the problem of selecting optimal compounds for first‐ and second‐line treatments based on prognostic factors, another primary goal is to determine the optimal time to initiate second‐line therapy, either immediately or delayed after induction therapy, yielding the longest overall survival time. A reinforcement learning method called Q‐learning is utilized, which involves learning an optimal regimen from patient data generated from the clinical reinforcement trial. Approximating the Q‐function with time‐indexed parameters can be achieved by using a modification of support vector regression that can utilize censored data. Within this framework, a simulation study shows that the procedure can extract optimal regimens for two lines of treatment directly from clinical data without prior knowledge of the treatment effect mechanism. In addition, we demonstrate that the design reliably selects the best initial time for second‐line therapy while taking into account the heterogeneity of NSCLC across patients.

Subfields: Pharmacology, Pulmonary and Respiratory Medicine, Statistics and Probability, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Pharmacology

In Pulmonary and Respiratory Medicine

In Statistics and Probability

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “ of the second and third lines of treatment vary from person to person. Developing valid methodology for computing optimal DTRs in such a ﬂexible setting is currently a premier challenge. Zhao et al. [88] presented an adaptive Q-learning approach to discover optimal DTRs for the ﬁrst and second line”
Estimating treatment effect heterogeneity in randomized program evaluation (2013, methodology): “…method can be applied to observational data [e.g., see Zhang et al. (2012) who develop a doubly robust estimator for optimal treatment regimes] and longitudinal data settings where the derivation of optimal dynamic treatment regimes is a frequent goal [e.g., Murphy (2003), Zhao et al. (2011)].”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, positive): “With the desired properties of dealing with delayed reward and inferring optimal policy based on non-optimal prescription behaviors, a set of reinforcement learning methods have been adapted to generate optimal DTR for life-threatening diseases, such as schizophrenia, non-small cell lung cancer, and”
A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units (2017, background): “Zhao et al. [2011] used Q-learning to learn optimal individualized treatment regimens for nonsmall cell lung cancer.”
Using Decision Lists to Construct Interpretable and Parsimonious Treatment Regimes (2015, methodology): “For nonparametric Q-learning, we use support vector regression when Y is continuous and support vector machines when Y is binary (Zhao et al., 2011), both are implemented using a Gaussian kernel.”
Interactive model building for Q-learning (2014, methodology): “…in the realistic case that the data-generating model is unknown; one approach would be to adopt nonparametric models for the Q-functions (e.g., Zhao et al., 2011; Moodie et al., 2013), but some clinicians are wary of black-box approaches and it 125 can be difficult to glean scientific…”
Q-LEARNING WITH CENSORED DATA (2012, methodology): “The approach closest to our proposal is the censored-Q-learning algo-rithm of Zhao et al. (2011).”
A Theoretical Analysis of Deep Q-Learning (2019, background): “rules for individual patients that adapt to time-evolving illnesses. There is a huge body of literature on this line of research. See, e.g., Murphy (2003); Zhao et al. (2009); Qian and Murphy (2011); Zhao et al. (2011); Zhang et al. (2012); Zhao et al. (2012); Goldberg and Kosorok (2012); Nahum-Shan”
Doubly Robust Learning for Estimating Individualized Treatment with Censored Data (2014, unknown): “Thus, there is increasing interest in dynamic treatment regimens (Thall et al., 2002; Murphy, 2003; Robins, 2004; Moodie et al., 2007; Zhao et al., 2011; Goldberg & Kosorok, 2012; Huang et al., 2014).”
New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes (2015, methodology): “Methods have been developed for the multi-decision problem within the Q-learning framework, where survival times are the outcome of interest (Zhao et al., 2011; Goldberg and Kosorok, 2012).”
Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection (2018, background): “lines of treatment for non-small cell lung cancer (Zhao et al., 2011).”
Machine learning in cardiovascular medicine: are we there yet? (2018, methodology): “39 Because many problems in clinical medicine can be formatted to fit the format of a reinforcement learning problem, we envision that this method will be used, for example, to select personalised blood pressure drugs and dosages, intelligently segment medical imaging data, or to interview patients…”
Precision Medicine. (2019, background): “Examples include the treatment of small-cell lung cancer where two or more lines of chemotherapy may be required (Zhao et al 2011).”
Doubly Robust Estimation of Optimal Dynamic Treatment Regimes (2013, unknown): “[21, 22].”
Adaptive Interventions Treatment Modelling and Regimen Optimization Using Sequential Multiple Assignment Randomized Trials (Smart) and Q-Learning (2018, methodology): “implemented Qlearning for learning an optimal regimen using training data generated from clinical trials assigned to patients with Non-Small Cell Lung Cancer [7].”
Residual Weighted Learning for Estimating Individualized Treatment Rules (2015, background): “However, since the data used for estimating the ITRs should not also be used for confirmation, it is expected that an additional randomized clinical trial comparing the estimated ITR with standard of care, or some other suitable treatment comparison, will be used for confirmation as is typically don”
Diagnostic Inferencing via Improving Clinical Concept Extraction with Deep Reinforcement Learning: A Preliminary Study (2017, background): “…learning for related clinical decision support tasks by mainly focusing on other modalities e.g. medical imaging (Netto et al., 2008) or specific domain-dependent use cases and clinical trials (Poolla, 2003; Shortreed et al., 2011; Zhao et al., 2011), but not for inferencing diagnosis.”