Foundation models for generalist medical artificial intelligence
By Michael Moor, Oishi Banerjee, Zahra F H Abad, H. Krumholz, Jure Leskovec, Eric J. Topol, Pranav Rajpurkar. Nature, 2023
1,871 citations recorded by the source corpus.
The exceptionally rapid development of highly flexible, reusable artificial intelligence (AI) models is likely to usher in newfound capabilities in medicine. We propose a new paradigm for medical AI, which we refer to as generalist medical AI (GMAI). GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities. Here we identify a set of high-impact potential applications for GMAI and lay out specific technical capabilities and training datasets necessary to enable them. We expect that GMAI-enabled applications will challenge current strategies for regulating and validating AI devices for medicine and will shift practices associated with the collection of large medical datasets. This review discusses generalist medical artificial intelligence, identifying potential applications and setting out specific technical capabilities and training datasets necessary to enable them, as well as highlighting challenges to its implementation.
Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging
View Paper: https://www.semanticscholar.org/paper/9faa2b0e5cb93f20df0555c3c350fab0b2eccf3a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Quoc V. Le
- Yoshua Bengio
- Jianfeng Gao
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
- Caiming Xiong
In Health Informatics
In Radiology, Nuclear Medicine and Imaging
- Pranav Rajpurkar
- Matthew P. Lungren
- Alan L. Yuille
- Jure Leskovec
- Dimitris N. Metaxas
- Shaoting Zhang
- Bradley Green
- Simon Kornblith
Citations in the corpus
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, background): “However, in real-world scenarios, LLMs tend to encounter domain-specific tasks that ∗ Equal contribution. necessitate a combination of domain expertise and complex reasoning abilities (Moor et al., 2023; Wu et al., 2023b; Singhal et al., 2023a; Yang et al., 2023).”
- On the Challenges and Perspectives of Foundation Models for Medical Image Analysis (2023, methodology): “They include classification, detection, segmentation and registration and utilize a single set of model weights Moor et al. [2023], Tu et al. [2023], Wu et al. [2023b].”
- Towards Generalist Biomedical AI (2023, background): “Generalist agents are not the only approach to multimodal biomedical AI While generalist biomedical AI systems offer exciting possibilities [43], there are other approaches to developing multimodal biomedical AI systems that might be more applicable depending on data availability, pretrained models,”
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, neutral): “…that leverage LLMs are increasingly enhancing the efficiency of numerous human activities, such as coding (Chen et al., 2021; Gao et al., 2023b), medical assistance (Yang et al., 2022; Moor et al., 2023), education (Kasneci et al., 2023; Kung et al., 2023), law (Katz et al., 2023), and so forth.”
- Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, methodology): “In the medical domain, this bears great potential to vastly expand the capabilities of existing medical AI models (Moor et al., 2023).”
- MetaboAnalyst 6.0: towards a unified platform for metabolomics data processing, analysis and interpretation (2024, background): “For future directions, we will continue to improve metabolome annotations, better integrate with other omics data, and explore new ways to interact with users via generative artificial intelligence technologies ( 70–73 ).”
- CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, methodology): “After we first presented CLIP-Driven Universal Model in arXiv and released the code, the field has witnessed numerous pivotal contributions [85, 47, 70, 28, 84, 5], with many more endeavors underway to our knowledge [56, 90].”
- BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “Biomedical data is inherently multimodal, comprising both physical measurements and natural-language narratives [1, 2].”
- A generalist vision–language foundation model for diverse biomedical tasks (2023, neutral): “However, most of today’s biomedical models act as specialist systems, tailored to specific tasks and modalities 2 .”
- Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions (2024, background): “We anticipate that such initiatives will significantly contribute to future advancements in medical foundation models and further propel the research community forward [88, 89].”