ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

By Kexin Huang, Jaan Altosaar, Rajesh Ranganath. 2019

1,329 citations recorded by the source corpus.

Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.

Subfields: Artificial Intelligence, Health Information Management

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Information Management

Citations in the corpus

SciBERT: A Pretrained Language Model for Scientific Text (2019, neutral): “Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “Other studies (e.g., Huang et al., 2019) have trained language models (LMs) in their domain of interest, from scratch.”
Towards Debiasing Sentence Representations (2020, background): “As their usage also proliferates across clinical applications [11, 1], it becomes necessary to recognize the role they play in shaping social biases and stereotypes.”
Pre-trained models for natural language processing: A survey (2020, background): “Recently, some studies have proposed PTMs trained on specialty corpora, such as BioBERT [96] for biomedical text, SciBERT [11] for scientific text, ClinicalBERT [68, 3] for clinical text.”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, background): “For example, Peng et al. (2019); Alsentzer et al. (2019); Huang et al. (2019) added clinical text to the PubMed biomedical pre-training corpus and tested on standard biomedical and clinical NLP benchmarks.”
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021, background): “, 2019), clinical (Huang et al., 2019) and legal NLP (Zheng et al.”
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation (2020, methodology): “Recently, some task-speciﬁc PrLMs have been proposed, such as BioBERT (Lee et al. 2020) for biomedical texts, SciBERT (Beltagy, Lo, and Cohan 2019) for scientiﬁc texts and Clinical-BERT (Huang, Altosaar, and Ranganath 2019) for clinical texts.”
Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, methodology): “This has resulted in the emergence of various models, including BioBERT (Lee et al., 2020), ClinicalBERT (Huang et al., 2019), PubMedBERT (Gu et al., 2021), BioLinkBERT (Yasunaga et al., b), DRAGON (Yasunaga et al., a), BioMedLM (Bolton et al., 2022), BioGPT (Luo et al., 2022), and Med-PaLM (Singhal”
A Survey on Vision Transformer (2020, methodology): “[55] proposes ClinicalBERT utilizing Transformer to develop and evaluate continuous representations of clinical notes and as a side effect, the attention map of ClinicalBERT can be used to explain predictions and thus discover high-quality connections between different medical contents.”
Train No Evil: Selective Masking for Task-guided Pre-training (2020, methodology): “To learn the domain-speciﬁc language patterns, some previous works (Beltagy et al., 2019; Huang et al., 2019) pre-train a BERT-like model from scratch using large-scale in-domain data.”
Large language models in healthcare and medical domain: A review (2023, methodology): “[122] T-5 (fine-tuned) model achieved the best ROGUE-L score of 24.85 .”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, background): “…endeavor to bridge the divide between pretraining domains and downstream domains has led to the adoption of a prevalent approach, known as continuous pretraining [4, 1, 15, 21], which involves the ongoing pretraining on an extensive collection of unlabeled data that is specific to a given domain.”
Hurtful words: quantifying biases in clinical contextual word embeddings (2020, methodology): “ich we use alongside the log probability score method to evaluate bias in the pretrained model. 2.3 Pretrained Clinical Embeddings Several BERT models pretrained on MIMIC notes are publicly available [3, 29, 60]. However, to address several limitations, we choose to train our own clinical BERT model”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “This is also displayed in the number of domain-adapted language models (Alsentzer et al., 2019; Huang et al., 2019; Adhikari et al., 2019; Lee and Hsiang, 2020; Jain and Ganesamoorty, 2020, (i.a.)), most notably BioBERT (Lee et al., 2020) that was pre-trained on biomedical PubMED articles and…”
On the Impact of Random Seeds on the Fairness of Clinical Classifiers (2021, methodology): “This vector is then used to calculate the final probability as P (Y = 1|Ỹ) = Ỹmax + ỸmeanN/c 1 +N/c , (1) where c is a scaling factor, which we set to c = 2, following Huang et al. (2019).”
TU_DBS in the ARQMath Lab 2021, CLEF (2021, methodology): “As a consequence, various models for different domains have been developed, such as BioBERT [12], ClinicalBERT [13, 14] or SciBERT [15] for scientific domains or CuBERT [16] and CodeBERT [17].”