PathVQA: 30000+ Questions for Medical Visual Question Answering

By Xuehai He, Yichen Zhang, Luntian Mou, Eric P. Xing, Pengtao Xie. arXiv.org, 2020

536 citations recorded by the source corpus.

Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a medical VQA dataset is much more challenging. First, due to privacy concerns, pathology images are usually not publicly available. Second, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. To address these challenges, we resort to pathology textbooks and online digital libraries. We develop a semi-automated pipeline to extract pathology images and captions from textbooks and generate question-answer pairs from captions using natural language processing. We collect 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. To our best knowledge, this is the first dataset for pathology VQA. Our dataset will be released publicly to promote research in medical VQA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fc0b46a0f3720e6c29c1a913aaa3de4a0699f713

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “In the existing data sources, there are 32k samples in PathVQA [46] and 12k in ScienceQA [84].”
A survey on multimodal large language models (2023, unknown): “LLaVA-Med [37] reports re-sults on biomedical VQA [69–71].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…ImageCLEF (en) [72], PMC (en) [261], SLAKE (en & zh) [145], GMAI-VL (en & zh) [134], VQA-Med (en) [15], Medical-Diff-VQA (en) [94], PathVQA (en) [83], Medical PMC-CaseReport (en) [260] Screen2Words (en) [240], WebSight (en) [122], Widget-Caption (en) [136], RICOSCA (en) [55], Seeclick (en)…”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “Finally, Med-Flamingo was evaluated on three VQA datasets (VQA-RAD [240], Path-VQA [245], Visual USMLE) with designed clinical evaluation score, BERT similarity score, and Exact-match score [244].”
FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding (2020, methodology): “In addition Path-VQA [33] and VQA-MED [2] are medical images for which VQA algorithms are also considered.”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “While discriminative methods yield good performance, they deal with closed-set predictions [14], and require mitigation when a customized answer set is provided in at inference [24, 51, 9].”