Med-Flamingo: a Multimodal Medical Few-shot Learner

By Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yashodhara Dalmia, E. Reis, Pranav Rajpurkar, Jure Leskovec. ML4H@NeurIPS, 2023

613 citations recorded by the source corpus.

Medicine, by its nature, is a multifaceted domain that requires the synthesis of information across various modalities. Medical generative vision-language models (VLMs) make a first step in this direction and promise many exciting clinical applications. However, existing models typically have to be fine-tuned on sizeable down-stream datasets, which poses a significant limitation as in many medical applications data is scarce, necessitating models that are capable of learning from few examples in real-time. Here we propose Med-Flamingo, a multimodal few-shot learner adapted to the medical domain. Based on OpenFlamingo-9B, we continue pre-training on paired and interleaved medical image-text data from publications and textbooks. Med-Flamingo unlocks few-shot generative medical visual question answering (VQA) abilities, which we evaluate on several datasets including a novel challenging open-ended VQA dataset of visual USMLE-style problems. Furthermore, we conduct the first human evaluation for generative medical VQA where physicians review the problems and blinded generations in an interactive app. Med-Flamingo improves performance in generative medical VQA by up to 20\% in clinician's rating and firstly enables multimodal medical few-shot adaptations, such as rationale generation. We release our model, code, and evaluation app under https://github.com/snap-stanford/med-flamingo.

View Paper: https://www.semanticscholar.org/paper/c9dbdae8146b9f97e254f5d26fd6efde96eaa703

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “Such M-LLMs with vision capabilities can be adapted for medical imaging applications (eg, LLaVA-Med [84], PMC-VQA [85], Med-Flamingo [86], and PeFoMed [87]).”
Large Language Models as Analogical Reasoners (2023, methodology): “This can include using reasoning-intensive data during training (Wu et al., 2021; Yasunaga et al., 2022b; Lightman et al., 2023; Moor et al., 2023), retrieving structured knowledge (Lin et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and…”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, background): “The implementation details is provided in the supplementary materials.”
A survey on multimodal large language models (2023, unknown): “The image is from [16].”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “22) Med-Flamingo: Med-Flamingo [196] is a vision-language model specifically designed to handle interleaved multimodal data comprising both images and text.”
HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, unknown): “…has been increased interest in their application in healthcare (Clusmann et al., 2023; Thirunavukarasu et al., 2023; Lee et al., 2023; Liu et al., 2023; Moor et al., 2023; Singhal et al., 2023; Omiye et al., 2024; Zakka et al., 2024; Nori et al., 2023; Cosentino et al., 2024; Shah et al., 2025).”
CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications. (2025, unknown): “Lastly, VLMs have been fine-tuned on scraped medical textbooks, 22 but without the use of IFT they are sometimes outperformed even by baseline models.”