Informing sequential clinical decision-making through reinforcement learning: an empirical study

By Susan M. Shortreed, Eric B. Laber, Daniel J. Lizotte, T. Scott Stroup, Joelle Pineau, Susan A. Murphy. 2010

223 citations recorded by the source corpus.

This paper highlights the role that reinforcement learning can play in the optimization of treatment policies for chronic illnesses. Before applying any off-the-shelf reinforcement learning methods in this setting, we must first tackle a number of challenges. We outline some of these challenges and present methods for overcoming them. First, we describe a multiple imputation approach to overcome the problem of missing data. Second, we discuss the use of function approximation in the context of a highly variable observation set. Finally, we discuss approaches to summarizing the evidence in the data for recommending a particular action and quantifying the uncertainty around the Q-function of the recommended policy. We present the results of applying these methods to real clinical trial data of patients with schizophrenia.

Subfields: Economics and Econometrics, Statistics and Probability, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “ivity in mouse brains in order to determine optimal stimulation frequencies for reducing epileptic seizures (Guez et al., 2008). Ofﬂine RL has also been used for optimizing long term treatment plans. Shortreed et al. (2011) uses ofﬂine ﬁtted Q-iteration for optimizing schizophrenia treatment, Nie et”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation (Prasad et al., 2017), sepsis management (Raghu et al., 2017a,b), and treatment of schizophrenia (Shortreed et al., 2011).”
Evaluating Reinforcement Learning Algorithms in Observational Health Settings (2018, methodology): “Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation [Prasad et al., 2017], sepsis management [Raghu et al., 2017] and treatment of schizophrenia [Shortreed et al., 2011].”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “an 18-month study divided into two main phases of treatment. An in-depth case study of using BRL, FQI, to optimize treatment choices for patients with schizophrenia using data from CATIE was given by [177]. Key technical challenges of applying RL in typically continuous, highly variable, and high-di”
Using Decision Lists to Construct Interpretable and Parsimonious Treatment Regimes (2015, methodology): “Indeed, decision lists have frequently been used to display estimated treatment regimes (Shortreed et al., 2011; Moodie et al., 2012; Shortreed et al., 2014; Laber and Zhao, 2015) or to describe theory-based, i.e., not data-driven, treatment regimes (Shiffman, 1997; Marlowe et al., 2012).”
Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach (2017, background): “Shortreed et al. (2011) learned optimal treatment policies for schizophrenic patients, and quantiﬁed the uncertainty around the expected outcome for patients who followed the policies.”
Artificial Intelligence in Cardiology. (2018, background): “(50) demonstrated that reinforcement learning can work for optimization of treatment policies in chronic illnesses.”
Transfer Learning Across Patient Variations with Hidden Parameter Markov Decision Processes (2016, background): “The use of RL (and machine learning, in general) in the development of optimal control policies and decision making strategies in healthcare [22] is gaining signiﬁcant momentum as methodologies have begun to adequately account for uncertainty and variations in the problem space.”
Mobile Health Technology Evaluation (2013, unknown): “In SMART, researchers decide which aspects of treatments require investigation and then randomize individuals at each treatment decision based on feasibility, ethical issues, or other factors.(25)”
Estimation of Optimal Dynamic Treatment Regimes (2014, background): “Optimal DTRs have been estimated from SMARTs for a wide range of chronic conditions including attention-deficit hyperactivity disorder [23,24], depression [25], HIV infection [13,26,27], schizophrenia [28], and cigarette addiction [5].”
Near-Optimal Model-Free Reinforcement Learning in Non-Stationary Episodic MDPs (2021, background): “…decision-making problems, such as online advertisement auctions (Cai et al., 2017; Lu et al., 2019), dynamic pricing (Chawla et al., 2016; Mao et al., 2018), trafﬁc management (Chen et al., 2020), health-care operations (Shortreed et al., 2011), and inventory con-trol (Agrawal & Jia, 2019).”
Diagnostic Inferencing via Improving Clinical Concept Extraction with Deep Reinforcement Learning: A Preliminary Study (2017, background): “…learning for related clinical decision support tasks by mainly focusing on other modalities e.g. medical imaging (Netto et al., 2008) or specific domain-dependent use cases and clinical trials (Poolla, 2003; Shortreed et al., 2011; Zhao et al., 2011), but not for inferencing diagnosis.”
The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making (2018, methodology): “[7] investigated fitted Q-iteration with linear function approximation to optimise treatment of schizophrenia.”
Model-Free Machine Learning in Biomedicine: Feasibility Study in Type 1 Diabetes (2016, background): “In medicine, RL is mainly investigated for prognosis, classification and diagnosis by means of big/heterogeneous data collection, fusion and analysis [25]-[29], with fewer reports on treatment studies [30], [31].”
Reinforcement Learning based Recommender Systems: A Survey (2021, methodology): “, PANSS) [90], and Jaccard distance between two states [44].”
Set-valued dynamic treatment regimes for competing outcomes (2012, methodology): “…ADHD (Nahum-Shani et al., 2010; Laber et al., 2011; Nahum-Shani et al., 2012), depression (Schulte et al., 2012; Song et al., 2012), HIV infection (Moodie, Richardson, and Stephens, 2007), schizophrenia (Shortreed et al., 2011), and cigarette addiction (Strecher, Shiffman, and West, 2006).”