Towards Generalist Biomedical AI

By Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, B. Mustafa and 21 more. NEJM AI, 2023

549 citations recorded by the source corpus.

Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system. Med-PaLM M is a large multimodal generative model that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. Med-PaLM M reaches performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. We also report examples of zero-shot generalization to novel medical concepts and tasks, positive transfer learning across tasks, and emergent zero-shot medical reasoning. To further probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist evaluation of model-generated (and human) chest X-ray reports and observe encouraging performance across model scales. In a side-by-side ranking on 246 retrospective chest X-rays, clinicians express a pairwise preference for Med-PaLM M reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.

View Paper: https://www.semanticscholar.org/paper/304f8b4edea01fdb5a2f7f8b998c83188deeccff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Challenges and Perspectives of Foundation Models for Medical Image Analysis (2023, methodology): “They include classification, detection, segmentation and registration and utilize a single set of model weights Moor et al. [2023], Tu et al. [2023], Wu et al. [2023b].”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “…photorealistic image generation (Esser et al., 2024; Saharia et al., 2022; Ramesh et al., 2021) have approached near-human capabilities, catalyzing major break-throughs in sectors such as medical imaging (Li et al., 2024; Tu et al., 2024) and autonomous driving (Xu et al., 2023; Jin et al., 2023a).”
Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024, unknown): “LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Adapted large language models can outperform medical experts in clinical text summarization (2023, unknown): “Recent data indicate that physicians can expend up to 2 hours on documentation for each hour of patient interaction 10 .”
A generalist vision–language foundation model for diverse biomedical tasks (2023, unknown): “In addition, we compared BiomedGPT-B with Med-PaLM M (12 billion parameters) using the weighted F 1 score, as reported in the paper 6 .”
A whole-slide foundation model for digital pathology from real-world data (2024, unknown): “Owing to their strong generalizability, foundation models have been developed for biomedical domains where labelled data are scarce but unlabelled data are abundant, a situation that aptly describes computational pathology 29–33 .”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “Biomedical data is inherently multimodal, comprising both physical measurements and natural-language narratives [1, 2].”
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, unknown): “3 3 Details about all guideline prompts and roles are shown in Section D for clarification.”
Towards accurate differential diagnosis with large language models (2023, background): “Furthermore, the integration of multimodal inputs by LLMs is an area of novel research [22, 23], with a large potential number of data modalities to consider, and little precedent for how information from multiple modalities should be integrated over time for a single case by AI systems.”