Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

By Wei Zhang, Xiaofeng He, Hongyuan Zha. 2018

295 citations recorded by the source corpus.

Dynamic treatment recommendation systems based on large-scale electronic health records (EHRs) become a key to successfully improve practical clinical outcomes. Prior relevant studies recommend treatments either use supervised learning (e.g. matching the indicator signal which denotes doctor prescriptions), or reinforcement learning (e.g. maximizing evaluation signal which indicates cumulative reward from survival rates). However, none of these studies have considered to combine the benefits of supervised learning and reinforcement learning. In this paper, we propose Supervised Reinforcement Learning with Recurrent Neural Network (SRL-RNN), which fuses them into a synergistic learning framework. Specifically, SRL-RNN applies an off-policy actor-critic framework to handle complex relations among multiple medications, diseases and individual characteristics. The "actor'' in the framework is adjusted by both the indicator signal and evaluation signal to ensure effective prescription and low mortality. RNN is further utilized to solve the Partially-Observed Markov Decision Process (POMDP) problem due to lack of fully observed states in real world applications. Experiments on the publicly real-world dataset, i.e., MIMIC-3, illustrate that our model can reduce the estimated mortality, while providing promising accuracy in matching doctors' prescriptions.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Reinforcement Learning, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Reinforcement Learning

In Medical & Health AI

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Wang et al. (2018) apply actor-critic methods on MIMIC-III to determine drug recommendations. dle such challenges may offer healthcare providers powerful assistive tools for optimizing the care of patients and ultimately improving outcomes.”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “[219] AC N/A DRL (DDPG) MIMIC-III Combining supervised and reinforcement learning for medication dosing covering a large number of diseases.”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “It has found applications in domains including robotic manipulation [25, 38, 48, 54], NLP [21, 22] and healthcare [52, 62].”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, methodology): “logy to alleviate the sample complexity burden in challenging reinforcement learning (RL) settings, particularly those where sample eciency is paramount [Mandel et al., 2014, Gottesman et al., 2018, Wang et al., 2018, Yu et al., 2019]. O-policy methods are often applied together with function approx”
Transformer Hawkes Process (2020, background): “Besides social media, event data also exist in domains like ﬁnancial transactions (Bacry et al., 2015) and personalized healthcare (Wang et al., 2018).”
Deep reinforcement learning for automated stock trading: an ensemble strategy (2020, unknown): “For future work, it will be interesting to explore more sophisticated model [44], solve empirical challenges [45], deal with large-scale data [46] such as S&P 500 constituent stocks.”
Deep Learning Based Recommender System (2017, methodology): “[164] proposed combining supervised deep reinforcement learning with RNNs for treatment recommendation.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “This paradigm is useful in a diverse array of application domains such as healthcare [51, 15, 35], autonomous driving [57, 4, 36], and recommendation systems [47, 12, 49].”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, background): “…complexity burden in challenging sequential decision making settings where sample-eﬃciency is crucial [Mandel et al., 2014, Gottesman et al., 2018, Wang et al., 2018, Yu et al., 2019]; it is seeing much recent interest due to the large amounts of oﬄine data already available in numerous…”
Reinforcement Learning based Recommender Systems: A Survey (2021, background): “The main idea in SRL-RNN [89] is to combine RL with supervised learning (SL) to recommend treatment based on electronic health records (EHRs).”