Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

By Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, Degui Zhi. npj Digital Medicine, 2020

1,033 citations recorded by the source corpus.

Deep learning (DL)-based predictive models from electronic health records (EHRs) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required by these models to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pretraining of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. Inspired by BERT, we propose Med-BERT, which adapts the BERT framework originally developed for the text domain to the structured EHR domain. Med-BERT is a contextualized embedding model pretrained on a structured EHR dataset of 28,490,650 patients. Fine-tuning experiments showed that Med-BERT substantially improves the prediction accuracy, boosting the area under the receiver operating characteristics curve (AUC) by 1.21–6.14% in two disease prediction tasks from two clinical databases. In particular, pretrained Med-BERT obtains promising performances on tasks with small fine-tuning training sets and can boost the AUC by more than 20% or obtain an AUC as high as a model trained on a training set ten times larger, compared with deep learning models without Med-BERT. We believe that Med-BERT will benefit disease prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/5d4de0fa45aeddc31142e6a24666d06ed7923f1e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “For such reason, we list detailed data usages and access links of each LLM in Section 3.2.”
Auditing large language models: a three-layered approach (2023, background): “LLMs can act as intermediary assets that are adapted for specific tasks, such as diagnosing medical conditions [66], generating code [67], [68] and translating languages [69].”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “It is highly used in disease prediction tasks, and it has given promising performances in disease prediction tasks and can boost the AUC by 20% [110] [111].”
Traj-CoA: Patient Trajectory Modeling via Chain-of-Agents for Lung Cancer Risk Prediction (2025, unknown): “Conventional approaches rely on task-specific feature engineering for specialized models, including recurrent neural networks [4, 5, 21], neural differential equations [6, 22, 23], and encoder-based transformers [24, 25, 26, 27].”
Towards Domain Specification of Embedding Models in Medicine (2025, unknown): “In the medical domain, transformer embeddings have been tailored to clinical data: Med-BERT [5] adds cross-visit pretraining on EHRs from Cerner and Truven MarketScan; ClinicalBERT [4] continues pretraining on MIMIC-III notes; BioBERT [2] augments pretraining with PubMed abstracts and PMC full…”