Pre-training of Graph Augmented Transformers for Medication Recommendation

By Junyuan Shang, Tengfei Ma, Cao Xiao, Jimeng Sun. International Joint Conference on Artificial Intelligence, 2019

352 citations recorded by the source corpus.

Medication recommendation is an important healthcare application. It is commonly formulated as a temporal prediction task. Hence, most existing works only utilize longitudinal electronic health records (EHRs) from a small number of patients with multiple visits ignoring a large number of patients with a single visit (selection bias). Moreover, important hierarchical knowledge such as diagnosis hierarchy is not leveraged in the representation learning process. Despite the success of deep learning techniques in computational phenotyping, most previous approaches have two limitations: task-oriented representation and ignoring hierarchies of medical codes. To address these challenges, we propose G-BERT, a new model to combine the power of Graph Neural Networks (GNNs) and BERT (Bidirectional Encoder Representations from Transformers) for medical code representation and medication recommendation. We use GNNs to represent the internal hierarchical structures of medical codes. Then we integrate the GNN representation into a transformer-based visit encoder and pre-train it on EHR data from patients only with a single visit. The pre-trained visit encoder and representation are then fine-tuned for downstream predictive tasks on longitudinal EHRs from patients with multiple visits. G-BERT is the first to bring the language model pre-training schema into the healthcare domain and it achieved state-of-the-art performance on the medication recommendation task.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/c3229debfda1b015c88404cf98f1074237d80809

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Self-Supervised Learning for Recommender Systems: A Survey (2022, methodology): “In the graph-based recommendation scenario, structure generation is formulated as: G-BERT [116] combines GNNs and BERT for medication recommendation.”
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020, result): “To the best of our knowledge, there are only two relevant studies in literature: BEHRT [43] and G-BERT [44].”
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive (2021, methodology): “Pre-training with self-supervised tasks and then using the pre-trained model for specific downstream tasks, either by fine-tuning or freezing the weights, is a common training strategy in graph SSL [82], [88], [89].”
Graph-Bert: Only Attention is Needed for Learning Graph Representations (2020, methodology): “FORMER and B ERT based learning approaches have been used extensively in various learning tasks [ Dai et al. , 2019; Lan et al. , 2019; Shang et al. , 2019 ] .”
Shifting machine learning for healthcare from development to deployment and from models to data (2022, methodology): “In view of their adeptness to model the sequential nature of clinical text, trans-formers have also been harnessed to model the sequential nature of clinical events 147–151 .”
Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer (2019, methodology): “There are recent works that use Transformer on medical records (Song et al. 2018; Wang and Xu 2019; Shang et al. 2019a; Li et al. 2019), but they either simply replace RNN with Transformer to handle ICU records, or directly apply BERT learning objective on medical records, and do not utilize the…”
Forecasting: theory and practice (2020, background): “…to assist diagnosis and disease progression (Ghassemi et al., 2015; Ma et al., 2017; Pierce et al., 2010; Qiao et al., 2019), to determine prognosis (Dietzel et al., 2010; Ng et al., 2007), and to recommend treatments for patients (Kedia and Williams, 2003; Scerri et al., 2006; Shang et al., 2019).”