Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

By Yifan Peng, Shankai Yan, Zhiyong Lu. 2019

995 citations recorded by the source corpus.

Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ ncbi-nlp/BLUE_Benchmark.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

GAN Memory with No Forgetting (2020, background): “Transfer learning Being general and effective, transfer learning has attracted increasing attention recently in various research fields, with a focus on discriminative tasks like classification [6, 70, 90, 93, 16, 24, 44, 46, 74, 92] and those in natural language processing [3, 63, 53, 52, 2].”
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020, methodology): “, BlueBERT (Peng et al., 2019) and BioBERT (Lee et al.”
CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT (2020, methodology): “Tclinical-rad is obtained by using Clinical BioBERT weight initializations (Alsentzer et al., 2019), which were obtained by further pretraining the BioBERT weights on 2 million clinical notes from the MIMIC-III database.”
A survey on sentiment analysis methods, applications, and challenges (2022, methodology): “ELMo contributes to over-coming the limitations of conventional word embedding approaches such as LSA, TF-IDF and n-grams models (Peng et al. 2019).”
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model (2023, methodology): “This modeling approach also influences medical language research(20) and is also evaluated.(23) Since then, bidirectional deep-learning models have been proposed like BERT(24) and Generative Pretrained Transformer (GPT).”
RadGraph: Extracting Clinical Entities and Relations from Radiology Reports (2021, methodology): “Biomedical pretraining For each of our approaches, in addition to using BERT weight initial-izations, we use weight initializations from four different biomedical pretrained models, which are BioBERT [35], Bio+ClinicalBERT [36], PubMedBERT [37], and BlueBERT [38].”
BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining (2022, methodology): “Speciﬁcally, [10] continue pre-training using PubMed abstracts and PubMed Central full text articles and [8] continue pre-training on both PubMed text and clinical notes from MIMIC-III [19].”
Assessment of DistilBERT performance on Named Entity Recognition task for the detection of Protected Health Information and medical concepts (2020, methodology): “Other BERT variations were also pre-trained on medical domain corpora such as BioBERT (Lee et al., 2019), BlueBERT (Peng et al., 2019) and ClinicalBERT (Alsentzer et al., 2019) that were pre-trained on PubMed which contains biomedical research articles and MIMIC-III which contains doctors’ notes…”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, methodology): “…tasks in the Biomedical and the Clinical domain (Luo et al., 2022; Lee et al., 2020; Alsentzer et al., 2019; Beltagy et al., 2019; Gu et al., 2020; Peng et al., 2019), NER across more datasets (Tjong Kim Sang and De Meulder, 2003; Malmasi et al., 2022; Fu et al., 2022; Laskar et al., 2022a (Wu et…”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[41], Academic [42]–[44] and Biomedical [45]–[48].”
ScholarBERT: Bigger is Not Always Better (2022, background): “ChemProt consists of 1820 PubMed abstracts with chemical-protein interactions annotated by domain experts (Peng et al., 2019).”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “BioBERT [91] FT PubMed, PMC Biomedical NER, RE, QA BlueBert [130] FT PubMed, MIMIC-III BLUE MIMIC-BERT [131] FT MIMIC-III Biomedical NER BioFLAIR [132] FT PubMed Bio NER Bio-ELECTRA-small [133] PT PubMed Biomedical NER AlphaBERT [134] FT Discharge diagnoses Extractive Summarization Task Spanish-bert”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, background): “For example, Peng et al. (2019); Alsentzer et al. (2019); Huang et al. (2019) added clinical text to the PubMed biomedical pre-training corpus and tested on standard biomedical and clinical NLP benchmarks.”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Best performance for the HoC dataset is obtained from (Peng et al., 2019).”
Self-Alignment Pretraining for Biomedical Entity Representations (2020, background): “B IO B ERT (Lee et al., 2020) and B LUE B ERT (Peng et al., 2019)) have seen many progress in biomedical text mining tasks.”
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2021, background): “, 2018), and similar initiatives in other domains (Peng et al., 2019), we introduce LexGLUE, a benchmark dataset to evaluate the performance of NLP methods in legal tasks.”