A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units

By Niranjani Prasad, Li-Fang Cheng, Corey J. Chivers, Michael Draugelis, Barbara E. Engelhardt. 2017

196 citations recorded by the source corpus.

The management of invasive mechanical ventilation, and the regulation of sedation and analgesia during ventilation, constitutes a major part of the care of patients admitted to intensive care units. Both prolonged dependence on mechanical ventilation and premature extubation are associated with increased risk of complications and higher hospital costs, but clinical opinion on the best protocol for weaning patients off of a ventilator varies. This work aims to develop a decision support tool that uses available patient information to predict time-to-extubation readiness and to recommend a personalized regime of sedation dosage and ventilator support. To this end, we use off-policy reinforcement learning algorithms to determine the best action at a given patient state from sub-optimal historical ICU data. We compare treatment policies from fitted Q-iteration with extremely randomized trees and with feedforward neural networks, and demonstrate that the policies learnt show promise in recommending weaning protocols with improved outcomes, in terms of minimizing rates of reintubation and regulating physiological stability.

Subfields: Artificial Intelligence, Critical Care and Intensive Care Medicine, Epidemiology, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Critical Care and Intensive Care Medicine

In Epidemiology

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Q-learning methods on this dataset has been applied to problems such as the treatment of sepsis (Raghu et al., 2017) and optimizing the use of ventilators (Prasad et al., 2017).”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation (Prasad et al., 2017), sepsis management (Raghu et al., 2017a,b), and treatment of schizophrenia (Shortreed et al., 2011).”
Evaluating Reinforcement Learning Algorithms in Observational Health Settings (2018, methodology): “Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation [Prasad et al., 2017], sepsis management [Raghu et al., 2017] and treatment of schizophrenia [Shortreed et al., 2011].”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “[220] applied off-policy RL algorithms, FQI-ERT and with feed”
Learning When-to-Treat Policies (2019, background): “…prevent AIDS while mitigating side effects (When To Start Consortium 2009), when to recommend mothers to stop breastfeeding to maximize infants’ health (Moodie, Platt, and Kramer 2009), and when to to turn off ventilators for intensive care patients to maximize health outcomes (Prasad et al. 2017).”
High-performance medicine: the convergence of human and artificial intelligence (2019, unknown): “129,130 ).”
Artificial intelligence, bias and clinical safety (2019, unknown): “…in medicine, but one can imagine a future where ‘closed loop’ applications, such as subcutaneous insulin pumps driven by information from wearable sensors, 33 or automated ventilator control driven by physiological monitoring data in intensive care, 34 are directly controlled by AI algorithms.”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, background): “, 2017), mechanical ventilation (Prasad et al., 2017), HIV (Ernst et al.”
Key challenges for delivering clinical impact with artificial intelligence (2019, background): “…length of stay and discharge diagnoses [34], predicting future deterioration, including acute kidney injury [35], improving decision-making strategies, including weaning of mechanical ventilation [36] and management of sepsis [37], and learning treatment policies from observational data [38].”
An Optimal Policy for Patient Laboratory Tests in Intensive Care Units (2018, unknown): “vector represents the estimated value of that state-action pair according to a dierent objective. We learn this Q-function using a variant of Fitted Q-iteration (FQI) with extremely randomized trees [13, 18]. FQI is a batch o-policy reinforcement learning algorithm that is well-suited to clinical ap”
Interpretable Batch IRL to Extract Clinician Goals in ICU Hypotension Management. (2020, background): “17 All these works take the reward as input and try to ﬁnd optimal treatment policies for the ICU.”
The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making (2018, methodology): “[8] compared different supervised learning methods to approximate action values, including neural networks, and successfully predicted the weaning of mechanical ventilation and sedation dosages.”