PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

By Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, Weidi Xie. arXiv.org, 2023

366 citations recorded by the source corpus.

Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery by leveraging artificial intelligence to interpret and answer questions based on medical images. In this study, we reframe the problem of MedVQA as a generation task that naturally follows the human-machine interaction and propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef-2019, significantly outperforming existing MedVQA models in generating relevant, accurate free-form answers. In addition, we propose a test set that has undergone manual verification, which is significantly more challenging, serving to better monitor the development of generative MedVQA methods. To facilitate comprehensive evaluation and comparison, we have maintained a leaderboard at https://paperswithcode.com/paper/pmc-vqa-visual-instruction-tuning-for-medical, offering a centralized resource for tracking progress and benchmarking state-of-the-art approaches. The PMC-VQA dataset emerges as a vital resource for the field of research, and the MedVInT presents a significant breakthrough in the area of MedVQA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f4793adffd6f67ffcb93ccfc5672ab301b8a2b96

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, methodology): “Existing medical VLMs include BiomedCLIP (Zhang et al., 2023a), MedVINT (Zhang et al., 2023b).”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “In addition, some efforts attempt to adapt MLLMs for domain-specific tasks, such as medical image understanding [15, 17, 50] and document analysis [20, 44].”
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, neutral): “3 3 Details about all guideline prompts and roles are shown in Section D for clarification.”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “A similar tradeoff exists between specialized models, such as PMC-VQA [147] for medical imagery, which achieve high accuracy in their niche, and general-purpose models like InstructBLIP, which exhibit broader zero-shot generalization to unseen tasks.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…Arxiv QA ALLaVA [25], SVIT [309], Cambrain-GPT4o [234], TextOCR-GPT4V [102], MMDU [159], Conversation Synthetic Real-World Conversations PMC-VQA [303], VQA-RAD [120], ImageCLEF [72], SLAKE [145], Medical-Diff-VQA [94], Medical PMC-CaseReport [260], GMAI-VL (subset) [134] GUI Screen2Words [240],…”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…to judge LMMs on skills that require external knowledge (Schwenk et al., 2022; Shah et al., 2019), common sense reasoning (Zellers et al., 2019; Yin et al., 2021), scientific-knowledge (Lu et al., 2022; Kembhavi et al., 2017, 2016), medical understanding (Zhang et al., 2023b; Lau et al., 2018).”
A survey on multimodal large language models (2023, neutral): “InstructBLIP [23], Ying-VLM [36], LLaVA-Med [37], MiniGPT-4 [13], X-LLM [25], MultiInstruct [26] GPT-aided PMC-VQA [30], VideoChat [33], DetGPT [38], LAMM [39], VisionLLM [24], MultiModal-GPT [31], LLaVA [21], ChatBridge [29], LLaVA-Med [37], GPT4Tools [34], Video-ChatGPT [40], Macaw-LLM [41]…”