A dataset of clinically generated visual questions and answers about radiology images

By J. J. Lau, Soumya Gayen, Asma Ben Abacha, Dina Demner-Fushman. Scientific Data, 2018

841 citations recorded by the source corpus.

Radiology images are an essential part of clinical decision making and population screening, e.g., for cancer. Automated systems could help clinicians cope with large amounts of images by answering questions about the image contents. An emerging area of artificial intelligence, Visual Question Answering (VQA) in the medical domain explores approaches to this form of clinical decision support. Success of such machine learning tools hinges on availability and design of collections composed of medical images augmented with question-answer pairs directed at the content of the image. We introduce VQA-RAD, the first manually constructed dataset where clinicians asked naturally occurring questions about radiology images and provided reference answers. Manual categorization of images and questions provides insight into clinically relevant tasks and the natural language to phrase them. Evaluating with well-known algorithms, we demonstrate the rich quality of this dataset over other automatically constructed ones. We propose VQA-RAD to encourage the community to design VQA tools with the goals of improving patient care. Design Type(s) image creation and editing objective • anatomical image analysis objective Measurement Type(s) image analysis Technology Type(s) visual observation method Factor Type(s) question type • answer type Sample Characteristic(s) Homo sapiens • head • chest • abdomen Design Type(s) image creation and editing objective • anatomical image analysis objective Measurement Type(s) image analysis Technology Type(s) visual observation method Factor Type(s) question type • answer type Sample Characteristic(s) Homo sapiens • head • chest • abdomen Machine-accessible metadata file describing the reported data (ISA-Tab format)

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/18f9a6045ba01cb079c4fa49a630d71bbd27cd92

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “In comparison, our proposed Vision-R1 demonstrates a substantial advantage in reasoning performance, effectively balancing CoT complexity and accuracy.”
PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? (2023, methodology): “However, MEVF is custom-tailored for the particular challenges encountered in the VQA-RAD (Lau et al., 2018) dataset, i.e., specifically designed for the organs present in this dataset, limiting its generalizability to other settings.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “Following, LLaVA-Med was evaluated on three biomedical medical visual question answering (VQA) datasets, VQA-RAD [240], SLAKE [241], and PathVQA [242].”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, methodology): “Dataset Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix database 0.”
What matters when building vision-language models? (2024, neutral): “…183,986 738,254 0.57% OK-VQA (Marino et al., 2019) 8,998 9,009 38,853 0.40% HatefulMemes (Kiela et al., 2020) 8,500 8,500 25,500 0.13% VQA-RAD (Lau et al., 2018 (Mathew et al., 2021) 10,189 39,463 337,829 3.46% TextCaps (Sidorov et al., 2020) 21,953 21,953 389,658 2.00% TextVQA (Singh et al.,…”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Interestingly, HELM pointed out that LMs such as T5 [95] and Anthropic-LMv4-s3 [5] had not been directly compared in the initial work, while LLMs such as GPT-3 and YaLM were still different from their corresponding reports after multiple evaluations.”
A survey on multimodal large language models (2023, neutral): “Traditional models typically allow a limited set of control mechanisms and often entail expensive curated datasets [103,104].”
PathVQA: 30000+ Questions for Medical Visual Question Answering (2020, methodology): “In VQA-Med, the authors collect medical images along with associated side information (e.g., captions, modalities, planes) from the MedPix 1 database and generate question-answer pairs based on manually-deﬁned patterns in [7].”
Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering (2021, background): “Comparison of SLAKE with VQA-RAD.”
Answering Questions about Data Visualizations using Efficient Bimodal Fusion (2019, background): “Several datasets for VQA has been proposed in recent years, include natural image understanding [27, 6], counting [1], reasoning about synthetic scenes [13], medical image analysis [25], scene text understanding [35], and video comprehension [12].”
Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain? (2021, background): “Lau et al. Lau et al. [2018] published VQA-RAD as the ﬁrst public benchmark dataset comprising realistic and clinically relevant question–answer pairs generated by expert clinicians and radiologists.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…to judge LMMs on skills that require external knowledge (Schwenk et al., 2022; Shah et al., 2019), common sense reasoning (Zellers et al., 2019; Yin et al., 2021), scientific-knowledge (Lu et al., 2022; Kembhavi et al., 2017, 2016), medical understanding (Zhang et al., 2023b; Lau et al., 2018).”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “We evaluated our method and competing methods on VQA-RAD [50].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…[25], SVIT [309], Cambrain-GPT4o [234], TextOCR-GPT4V [102], MMDU [159], Conversation Synthetic Real-World Conversations PMC-VQA [303], VQA-RAD [120], ImageCLEF [72], SLAKE [145], Medical-Diff-VQA [94], Medical PMC-CaseReport [260], GMAI-VL (subset) [134] GUI Screen2Words [240], WebSight [122]…”
A generalist vision–language foundation model for diverse biomedical tasks (2023, neutral): “The VQA datasets included radiology data covering five anatomies (VQA-RAD 17 and Semantically-Labeled Knowledge-Enhanced Dataset (SLAKE) 18 ), in addition to pathology data that captures both anatomical and tissue-specific details (PathVQA 19 ).”
VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019 (2019, methodology): “Patterns : For each category, we selected question patterns from hundreds of questions naturally asked and validated by medical students from the VQA-RAD dataset [13].”