Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach

By Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, Marzyeh Ghassemi. 2017

238 citations recorded by the source corpus.

Sepsis is a leading cause of mortality in intensive care units (ICUs) and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. Understanding more about a patient's physiological state at a given time could hold the key to effective treatment policies. In this work, we propose a new approach to deduce optimal treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Learning treatment policies over continuous spaces is important, because we retain more of the patient's physiological information. Our model is able to learn clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. Evaluating our algorithm on past ICU patient data, we find that our model could reduce patient mortality in the hospital by up to 3.6% over observed clinical policies, from a baseline mortality of 13.7%. The learned treatment policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival.

Subfields: Artificial Intelligence, Critical Care and Intensive Care Medicine, Epidemiology, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Critical Care and Intensive Care Medicine

In Epidemiology

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, methodology): “The authors thank the EuResist Network for providing HIV data and Matthieu Komorowski for sepsis data (Raghu et al. 2017).”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “1 As pointed out by Raghu et al. (2017a), RL also has advantages over other machine learning algorithms even in the observational setting, especially in two situations: when the optimal treatment strategy is unclear in medical literature (Marik, 2015), and when training examples do not represent…”
Reinforcement Learning in Healthcare: A Survey (2019, background): “[194], [195] examined fully continuous state and action space, where policies are learned directly from the physiological state data.”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “…advertising and marketing [Bottou et al., 2013; Tang et al., 2013; Chapelle et al., 2015; Theocharous et al., 2015; Thomas et al., 2017] or medical treatments [Murphy et al., 2001; Ernst et al., 2006; Raghu et al., 2017], online policy evaluation is usually expensive, risky, or even unethical.”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, neutral): “The ability to characterize the bias of a deep generative model is an important step towards using these models to guide decisions in high-stakes applications under uncertainty [53, 54], such as healthcare [55–57] and robust anomaly detection [58, 59].”
Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients (2017, unknown): “RL also has the ability to generate optimal strategies based on non-optimized training data [Raghu et al., 2017].”
Machine learning in critical care: state-of-the-art and a sepsis case study (2018, background): “[57] for the definition of continuous-space models for sepsis treatment, in a twist that goes beyond the more traditional development and use of discriminative classifiers.”
Deep Inverse Reinforcement Learning for Sepsis Treatment (2019, methodology): “In the DIRL component, Dueling Double Deep Q Network (DDDQN) [6], Bayesain inverse RL [7], [8] and Stochastic gradient descent [9], [10] methods are applied to infer the optimal reward function, in terms of optimal weights in these key influencing factors, based on treatment actions of the clinician”
Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks (2019, background): “…(EHRs) in modern healthcare systems has enabled the secondary use of these records to develop machine learning models for mortality risk (Harutyunyan et al., 2017), sepsis treatment (Raghu et al., 2017), and many other promising applications (Lim and van der Schaar, 2018; Rajkomar et al., 2018).”
Model-Based Reinforcement Learning for Sepsis Treatment (2018, background): “Qualitative analysis of learned treatment policies (as seen in previous work by [15]) does not necessarily provide an absolute indicator of performance, explored in more detail by [4].”
Towards High Confidence Off-Policy Reinforcement Learning for Clinical Applications (2018, methodology): “Raghu et al. (2017) use Q-learning to optimize policy for sepsis treatment.”
Dynamic Measurement Scheduling for Adverse Event Forecasting using Deep RL (2018, methodology): “Deep RL in health care Several recent works used RL to learn a treatment plan in ICU (Weng et al., 2017; Raghu et al., 2017; Futoma et al., 2018).”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “In this case, we may still try to teach an agent a good policy, using the observed sequences of actions/states/rewards in conjunction with oﬀ-policy methods [209, 182, 134], but thorough validation and evaluation can be challenging.”
Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters (2018, methodology): “We use the same framing as Raghu et al. (2017), where the medical treatment process for a sepsis patient is framed as a continuous state-space MDP.”
Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection (2018, background): “In the health context, temporal data in the form of electronic health records and intensive care unit interventions over time have both been represented in an MDP’s state (Nemati et al., 2016; Raghu et al., 2017; Prasad et al., 2017).”