Evaluating Reinforcement Learning Algorithms in Observational Health Settings

By Omer Gottesman, Fredrik D. Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, David Wihl, Xuefeng Peng, Jiayu Yao and 8 more. 2018

151 citations recorded by the source corpus.

Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Toward Causal Representation Learning (2021, background): “However, as soon as off-policy learning is considered, in particular, in the batch (or observational) setting [146], issues of causality become subtle [82], [165].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Using offline reinforcement learning in healthcare poses several unique challenges (Gottesman et al., 2018).”
Towards Causal Representation Learning (2021, background): “off-policy learning is considered, in particular in the batch (or observational) setting [146], issues of causality become subtle [164, 81].”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “After training the D3QN on the augmented data set, we estimated the Q-values of the physician policy as well as that of the learned optimal policy (Q-values have been typically used to evaluate patient care, e.g., [24, 3, 4]).”
Causality for Machine Learning (2019, background): “However, as soon as off-policy learning is considered, in particular in the batch (or observational) setting (Lange et al., 2012), issues of causality become subtle (Lu et al., 2018; Gottesman et al., 2018).”
Deconfounding Reinforcement Learning in Observational Settings (2018, background): “The reason for this is that we do not wish to experiment with patients’ lives without evidence that the proposed treatment strategy is better than the current practice (Gottesman et al., 2018).”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “Using sepsis management as an example, Gottesman et al. [72] discussed in detail why evaluation of polices using retrospective healthcare data is a fundamentally challenging issue.”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
A Minimalist Approach to Offline Reinforcement Learning (2021, background): “This questions the empirical effectiveness of offline RL for safety-critical real-world use cases [Mandel et al., 2014, Gottesman et al., 2018, Gauci et al., 2018, Jaques et al., 2019, Matsushima et al., 2020] as well as the current trend of reporting only the mean-value of the final policy in…”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…RL has shown promising results for a variety of real-world applications, such as robotics (Mandlekar et al., 2019), dialogue systems (Jaques et al., 2019), or medical treatments (Gottesman et al., 2018), these algorithms struggle when learning a policy from scratch or when the dataset is small.”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, unknown): “…off-policy methods or batch methods) are a promising methodology to alleviate the sample complexity burden in challenging reinforcement learning (RL) settings, particularly those where sample efficiency is paramount [Mandel et al., 2014, Gottesman et al., 2018, Wang et al., 2018, Yu et al., 2019].”
Deep Reinforcement Learning for Closed-Loop Blood Glucose Control (2020, background): “Oﬀ-policy RL with function approximation, particularly deep neural networks, is known to be unstable (Sutton and Barto, 2018; Gottesman et al., 2018).”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…to alleviate the sample complexity burden in challenging sequential decision making settings where sample-eﬃciency is crucial [Mandel et al., 2014, Gottesman et al., 2018, Wang et al., 2018, Yu et al., 2019]; it is seeing much recent interest due to the large amounts of oﬄine data already…”
Model-Based Reinforcement Learning for Sepsis Treatment (2018, background): “Qualitative analysis of learned treatment policies (as seen in previous work by [15]) does not necessarily provide an absolute indicator of performance, explored in more detail by [4].”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, methodology): “Following previous works (Petersen et al., 2018; Gottesman et al., 2018), we focus on the unsupervised dynamics generalization setting, i.e. the id or label information of dynamics function in training MDPs is not available.”