Publicly Available Clinical BERT Embeddings

By Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, Matthew B. A. McDermott. 2019

2,728 citations recorded by the source corpus.

Contextual word embedding models such as ELMo and BERT have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on 3/5 clinical NLP tasks, establishing a new state-of-the-art on the MedNLI dataset. We find that these domain-specific models are not as performant on 2 clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SciBERT: A Pretrained Language Model for Scientific Text (2019, neutral): “Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “We initialize our encoder with the ClinicalBERT weights (Alsentzer et al., 2019) pretrained on the MIMIC clinical notes, which achieved state-of-the-art performance on a suite of clinical NLP tasks.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “Transfer learning for domain adaptation Prior work has shown the benefit of continued pretraining in domain (Alsentzer et al., 2019; Chakrabarty et al., 2019; Lee et al., 2019).”
Towards Debiasing Sentence Representations (2020, background): “As their usage prolifer-ates across various real-world applications (Huang et al., 2019; Alsentzer et al., 2019), it becomes necessary to recognize the role they play in shaping social biases and stereotypes.”
Pre-trained models for natural language processing: A survey (2020, background): “Recently, some studies have proposed PTMs trained on specialty corpora, such as BioBERT [98] for biomedical text, SciBERT [11] for scientific text, ClinicalBERT [69, 3] for clinical text.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “MTClinicalBERT achieved on par but less performance compared to task-specific ClinicalBERT models.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, methodology): “In the report encoder we use the BERT base PyTorch implementation from Huggingface Transformers[84]10 and initialize it with weights from ClinicalBERT[1]11 that was trained on clinical notes.”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, background): “In biomedical NLP, this has often meant pretraining models over collections of Electronic Health Records (EHRs) (Alsentzer et al., 2019).”
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare (2021, methodology): “, 2020) and clinical BERT (Alsentzer et al., 2019; Huang et al., 2019) for the biomedical and clinical domain, respectively.”
LEGAL-BERT: “Preparing the Muppets for Court’” (2020, methodology): “suggestions are blindly followed in the literature (Lee et al., 2019; Alsentzer et al., 2019; Beltagy et al., 2019; Sung et al., 2019).”
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission (2019, methodology): “Concurrent to our work, Lee et al. (2019) apply bert to biomedical literature and Alsentzer et al. (2019) apply bert on clinical notes and discharge summaries.”
Hurtful words: quantifying biases in clinical contextual word embeddings (2020, methodology): “ich we use alongside the log probability score method to evaluate bias in the pretrained model. 2.3 Pretrained Clinical Embeddings Several BERT models pretrained on MIMIC notes are publicly available [3, 29, 60]. However, to address several limitations, we choose to train our own clinical BERT model”
Multimodal Representation Learning via Maximization of Local Mutual Information (2021, methodology): “We use the clinical BERT model [1] as the text encoder for both reportlevel and sentence-level feature extraction.”
Continual Pre-training of Language Models (2023, background): “DAP-training can be achieved by directly updating the LM (Xu et al., 2019; Sun et al., 2019; Lee et al., 2020; Alsentzer et al., 2019; Gururangan et al., 2020; Chakrabarty et al., 2019; Ke et al., 2022b) or by training only a small set of additional parameters.”
CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT (2020, methodology): “Tclinical-rad is obtained by using Clinical BioBERT weight initializations (Alsentzer et al., 2019), which were obtained by further pretraining the BioBERT weights on 2 million clinical notes from the MIMIC-III database.”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, background): “For example, Peng et al. (2019); Alsentzer et al. (2019); Huang et al. (2019) added clinical text to the PubMed biomedical pre-training corpus and tested on standard biomedical and clinical NLP benchmarks.”
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020, methodology): “BERT also has been applied to the clinical domain, where there already were publicly available models pre-trained on clinical texts [33,34,40].”