Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

By Yu Gu, Robert Tinn, Hao Cheng, Michael R. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon. 2020

2,598 citations recorded by the source corpus.

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, methodology): “PubMedBERT [11] is an encoder-based BERT-like model that is trained from scratch using abstracts from PubMed and full-text articles from PubMedCentral in the corpus “The Pile” [10].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, positive): “For the text data, we use PubMed (pub), a widely-used corpus in biomedial LM training (e.g., BioBERT (Lee et al. 2020), PubmedBERT (Gu et al. 2020)).”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, methodology): “BERTBase & Large We compare our models to the pre-trained BERTBase & Large models of BioBERT (Lee et al., 2019) and PubMedBERT (Gu et al., 2020) (BERTBase) for fine-tuning and evaluation.”
Self-Alignment Pretraining for Biomedical Entity Representations (2020, methodology): “P UB M ED B ERT (Gu et al., 2020)) model as suggested in Fig.”
RadGraph: Extracting Clinical Entities and Relations from Radiology Reports (2021, methodology): “Biomedical pretraining For each of our approaches, in addition to using BERT weight initializations, we use weight initializations from four different biomedical pretrained models, which are BioBERT [33], Bio+ClinicalBERT [34], PubMedBERT [35], and BlueBERT [36].”
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents (2023, methodology): “For the visual and text encoders, we adopt ResNet50 [12] and PubmedBERT [11].”
TimeLMs: Diachronic Language Models from Twitter (2022, background): “A well-known case is the biomedical domain, e.g., BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019) or PubMedBERT (Gu et al., 2021).”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Examples of biomedical text LLMs include BiMediX [104], BioBERT [105], PubMedBERT [106], and ClinicalBERT [20].”
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, background): “Although previous research has shown that language models pre-trained with biomedical domain specific data can benefit various in-domain downstream tasks [37-39], pre-training a language model from scratch or existing checkpoint can be very costly, especially when the sizes of language models growin”
LinkBERT: Pretraining Language Models with Document Links (2022, methodology): “For the biomedical domain, LinkBERT exceeds PubmedBERT (Gu et al., 2020) and attains new state-of-the-art on BLURB biomedical NLP benchmark (+3% absolute in BLURB score) and MedQA-USMLE reasoning task (+7% absolute in accuracy).”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Several studies [125], [129], [131], [132] explore the utilities and constraints of LLMs such as ChatGPT in the context of clinical practice, research, and public health.”
Large language models are few-shot clinical information extractors (2022, methodology): “PubMedBERT with an additional classification layer (LSTM or CRF) achieves close to state-of-the-art performance on the full task (Gu et al., 2021).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “ferase”] by SciBERT which has domain-specific vocabulary [47].”
Large language models encode clinical knowledge (2022, background): “LLMs for science and biomedicine Recent studies, such as SciBERT [5], BioNLP [46], BioMegatron [76], BioBERT [44], PubMedBERT [25], DARE [66], ScholarBERT [31], and BioGPT [56], have demonstrated the effectiveness of using curated scientific and biomedical corpora for both discriminative and generat”