Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

By Chao Yu, Jiming Liu, Hongyi Zhao. 2019

61 citations recorded by the source corpus.

BackgroundReinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. To ensure such applications, an explicit reward function encoding domain knowledge should be specified beforehand to indicate the goal of tasks. However, there is usually no explicit information regarding the reward function in medical records. It is then necessary to consider an approach whereby the reward function can be learned from a set of presumably optimal treatment trajectories using retrospective real medical data. This paper applies inverse RL in inferring the reward functions that clinicians have in mind during their decisions on weaning of mechanical ventilation and sedative dosing in Intensive Care Units (ICUs).MethodsWe model the decision making problem as a Markov Decision Process, and use a batch RL method, Fitted Q Iterations with Gradient Boosting Decision Tree, to learn a suitable ventilator weaning policy from real trajectories in retrospective ICU data. A Bayesian inverse RL method is then applied to infer the latent reward functions in terms of weights in trading off various aspects of evaluation criterion. We then evaluate how the policy learned using the Bayesian inverse RL method matches the policy given by clinicians, as compared to other policies learned with fixed reward functions.ResultsResults show that the inverse RL method is capable of extracting meaningful indicators for recommending extubation readiness and sedative dosage, indicating that clinicians pay more attention to patients’ physiological stability (e.g., heart rate and respiration rate), rather than oxygenation criteria (FiO2, PEEP and SpO2) which is supported by previous RL methods. Moreover, by discovering the optimal weights, new effective treatment protocols can be suggested.ConclusionsInverse RL is an effective approach to discovering clinicians’ underlying reward functions for designing better treatment protocols in the ventilation weaning and sedative dosing in future ICUs.

Subfields: Artificial Intelligence, Epidemiology, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chao Yu
2. Jiming Liu
3. Shamim Nemati
4. Guoqi Ren
5. Yinzhao Dong

In Epidemiology

1. Chao Yu
2. Guoqi Ren
3. Jiming Liu
4. Shamim Nemati
5. Yinzhao Dong

In Health Informatics

1. Guoqi Ren
2. Chao Yu
3. Shamim Nemati
4. Yinzhao Dong
5. Jiming Liu

Citations in the corpus

• Reinforcement Learning in Healthcare: A Survey (2019, methodology): “[219] Q-learning BRL(FQI) IRL MIMIC-III Applying IRL in inferring the reward functions.”
• Deep Inverse Reinforcement Learning for Sepsis Treatment (2019, methodology): “In the DIRL component, Dueling Double Deep Q Network (DDDQN) [6], Bayesain inverse RL [7], [8] and Stochastic gradient descent [9], [10] methods are applied to infer the optimal reward function, in terms of optimal weights in these key influencing factors, based on treatment actions of the clinician”
• Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units (2020, methodology): “Please refer to [19] for more details in data processing.”