A generalist vision–language foundation model for diverse biomedical tasks

By Zhiling Yan, Eashan Adhikarla, S. Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, Lifang He, B. Davison, Quanzheng Li and 3 more. Nature Medicine, 2023

401 citations recorded by the source corpus.

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners and patients. Here, we describe BiomedGPT, the first open-source and lightweight vision–language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency. An open-source and computing-friendly vision–language model achieves state-of-the-art accuracy in 16 out of 25 biomedical tasks, with promising performance in a series of potential clinical applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2ad26fd9ac2f88ac6705842b665f2f5c12cac108

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Generalist Biomedical AI (2023, methodology): “BiomedGPT [46] is a multi-task biomedical foundation model pretrained on a diverse source of medical images, medical literature, and clinical notes using a combination of language model (LM) and masked image infilling objectives.”
“Small” Large Language Models in the Hospital: Evaluation Study on Real-World Data in a Resource-Constrained Setting (2025, unknown): “Similarly, BioMedGPT-LM-7B [17] was excluded because, despite its domain-specific fine-tuning, it lacked instruction tuning necessary for the intended applications.”