BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

By Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, Tie-Yan Liu. Briefings Bioinform., 2022

1,456 citations recorded by the source corpus.

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/44279244407a64431810f982be6d0c7da4429dd7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Towards Expert-Level Medical Question Answering with Large Language Models (2023, neutral): “This is below the state-of-the-art performance (81.0 from BioGPT-Large [15]) and is likely because no data was included for this dataset for instruction ﬁnetuning.”
Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, methodology): “…in the emergence of various models, including BioBERT (Lee et al., 2020), ClinicalBERT (Huang et al., 2019), PubMedBERT (Gu et al., 2021), BioLinkBERT (Yasunaga et al., b), DRAGON (Yasunaga et al., a), BioMedLM (Bolton et al., 2022), BioGPT (Luo et al., 2022), and Med-PaLM (Singhal et al., 2022).”
Welcome to the Era of ChatGPT et al. (2023, background): “Note, however, that ﬁrst implementations of domain-speciﬁc models (e.g., BioGPT; for the biomedical literature) are already emerging (Luo et al. 2023).”
A study of generative large language model for medical research and healthcare (2023, positive): “We compared the two GatorTronGPT models with four existing transformer models including GPT-2 43 , REBEL, REBEL-pt 25 , and BioGPT 18 on three biomedical tasks for end-to-end relation extraction using three benchmark datasets including drug-drug interaction 44 (DDI), BioCreative V chemical-disease…”
BloombergGPT: A Large Language Model for Finance (2023, background): “The few existing domain-specific LLMs are trained exclusively on domain-specific data sources (Luo et al., 2022; Bolton et al., 2023; Taylor et al., 2022), or adapt a very large general purpose model to domain-specific tasks (Singhal et al., 2022; Lewkowycz et al., 2022).”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, methodology): “As aligned Large Language Models (LLMs) have been widely used to support decision-making in both professional and social domains (Araci, 2019; Luo et al., 2022; Tinn et al., 2023), they have been equipped with safety features that can prevent them from generating harmful or objection-able responses…”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “We have previously used PubMed papers to pretrain state-of-the-art biomedical large language models (e.g., PubMedBERT [20], BioGPT [21]).”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, methodology): “For instance, tasks in the Biomedical and the Clinical domain (Luo et al., 2022; Lee et al., 2020; Alsentzer et al., 2019; Beltagy et al., 2019; Gu et al., 2020; Peng et al., 2019), NER across more datasets (Tjong Kim Sang and De Meulder, 2003; Malmasi et al., 2022; Fu et al., 2022; Laskar et al.,…”
HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “Some works have focused on specialized medical models (Luo et al., 2022; Singhal et al., 2023; Li et al., 2023; Singhal et al., 2025; Saab et al., 2024), and others on general models (Nori et al., 2023; Gilson et al., 2023; Takagi et al., 2023).”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “BioGPT [110] is a large-scale language model that was constructed by the Allen Institute for AI (AI2) with the explicit purpose of undertaking training on biomedical text.”
Benchmarking Retrieval-Augmented Generation for Medicine (2024, background): “Due to their knowledge-intensive nature, QA datasets are commonly used to evaluate the biomedical capabilities of both general LLMs (Nori et al., 2023a,b) and domain-specific LLMs (Luo et al., 2022; Chen et al., 2023b; Wu et al., 2023; Singhal et al., 2023a,b).”