GRAM: Graph-based Attention Model for Healthcare Representation Learning

By Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, Jimeng Sun. 2016

770 citations recorded by the source corpus.

Deep learning methods exhibit promising performance for predictive modeling in healthcare, but two important challenges remain: - Data insufficiency: Often in healthcare predictive modeling, the sample size is insufficient for deep learning methods to achieve satisfactory results. Interpretation: The representations learned by deep learning methods should align with medical knowledge. To address these challenges, we propose GRaph-based Attention Model (GRAM) that supplements electronic health records (EHR) with hierarchical information inherent to medical ontologies. Based on the data volume and the ontology structure, GRAM represents a medical concept as a combination of its ancestors in the ontology via an attention mechanism. We compared predictive performance (i.e. accuracy, data needs, interpretability) of GRAM to various methods including the recurrent neural network (RNN) in two sequential diagnoses prediction tasks and one heart failure prediction task. Compared to the basic RNN, GRAM achieved 10% higher accuracy for predicting diseases rarely observed in the training data and 3% improved area under the ROC curve for predicting heart failure using an order of magnitude less training data. Additionally, unlike other methods, the medical concept representations learned by GRAM are well aligned with the medical ontology. Finally, GRAM exhibits intuitive attention behaviors by adaptively generalizing to higher level concepts when facing data insufficiency at the lower level concepts.

Subfields: Artificial Intelligence, Health Information Management, Representation & Self-Supervised Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Information Management

In Representation & Self-Supervised Learning

In Graph Learning

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, positive): “This is compatible with previous findings that inputting medical/domain relational knowledge has led to better out of domain behavior (Nestor et al., 2019), performance (Popescu and Khalilia, 2011; Choi et al., 2017; Tomašev et al., 2019a,b) and interpretability (Panigutti et al., 2020).”
Learning Credible Models (2017, background): “[5] looks at utilizing hierarchical expert information to learn embeddings that help model prediction of rare diseases.”
Interpretable Representation Learning for Healthcare via Capturing Disease Progression through Time (2018, methodology): “[7] proposes Gram model which uses attention mechanism to utilize domain knowledge about the medical codes.”
A Comprehensive Survey on Graph Neural Networks (2019, background): “…GNNs to a variety of problems such as program veriﬁcation [17], program reasoning [166], social inﬂuence prediction [167], adversarial attacks prevention [168], electrical health records modeling [169], [170], brain networks [171], event detection [172], and combinatorial optimization [173].”
Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review (2018, background): “In 42 , this is achieved by learning the latent embedding of a clinical code (eg., diagnosis code) as a convex combination of the embeddings of the code itself and its ancestors on the ontology graph.”
Capturing Feature-Level Irregularity in Disease Progression Modeling (2017, methodology): “In their “GRAM” model [8], they employ a graph-based attention model, and input EMR data of patients’ visits in chronological order without considering the intrinsic irregularity.”
Opportunities in Machine Learning for Healthcare (2018, neutral): “ning representations of other data types that simultaneously capture hierarchy and similarity (Nickel and Kiela, 2017), and recent work has examined using diagnostic code ontologies to aid prediction(Choi et al., 2017), and addingnodes to the hiddenlayer of neural networkmodels based on knownbiologi”
Explainable Recommendation through Attentive Multi-View Learning (2019, methodology): “We follow the method proposed in (Choi et al. 2017) to train feature embeddings that capture both semantic and hierarchical information.”
Personalized Prescription for Comorbidity (2018, methodology): “– initial-PPC (i-PPC): It is with the same setting as our model, except using GRAM [29] to learn the representation of diseases.”
Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks (2017, methodology): “RETAIN applies an RNN with reverse time ordered EHR sequences, while GRAM uses an RNN when modeling time ordered patient visits.”
Pre-training of Graph Augmented Transformers for Medication Recommendation (2019, methodology): “It is worth mentioning that our graph embedding method on medical ontology is different from GRAM [Choi et al. , 2017] from the following two aspects: 1.”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, background): “sification systems such as the International Classification of Diseases (ICD) [56], [63].”
Attention Models in Graphs (2018, methodology): “[Choi et al. 2017] uses an attention mechanism to learn a k -dimensional final embedding of each leaf node i (medical concept) via: where A( i ) denotes the set comprised of i and all its ancestors and e j is the k -dimensional basic embedding of the node j .”