Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering

By Bo Liu, Li-Ming Zhan, Li Xu, Lingxing Ma, Y. Yang, Xiao-Ming Wu. IEEE International Symposium on Biomedical Imaging, 2021

635 citations recorded by the source corpus.

Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the development and evaluation of Med-VQA systems. The dataset can be downloaded from http://www.med-vqa.com/slake.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/93b6b79b4ef6c345f31722ce7c829385c6dce0d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain? (2021, background): “Vision Transformers indeed are capable of accounting for such features Yu et al. [2021], and hence perform better on the SLAKE dataset.”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Interestingly, HELM pointed out that LMs such as T5 [95] and Anthropic-LMv4-s3 [5] had not been directly compared in the initial work, while LLMs such as GPT-3 and YaLM were still different from their corresponding reports after multiple evaluations.”
A generalist vision–language foundation model for diverse biomedical tasks (2023, positive): “Additionally, we selected MRI-only and CT-only image–text pairs from SLAKE and conducted VQA fine-tuning.”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, background): “MedVQA is a task of answering natural language questions about medical visual content, typically images or videos obtained from medical devices like X-ray, CT, MRI, or microscopy, etc. Specifically, our goal is to train a model that can output the corresponding answer for a given question, which…”
A survey on multimodal large language models (2023, neutral): “LLaVA-Med [37] reports re-sults on biomedical VQA [69–71].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…[234], TextOCR-GPT4V [102], MMDU [159], Conversation Synthetic Real-World Conversations PMC-VQA [303], VQA-RAD [120], ImageCLEF [72], SLAKE [145], Medical-Diff-VQA [94], Medical PMC-CaseReport [260], GMAI-VL (subset) [134] GUI Screen2Words [240], WebSight [122] Type: Video Datasets…”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “SLAKE (English only) [51] [25], where all previous VQA models failed to produce the correct answer (including PubMedCLIP, the prior state of the art on this task).”
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents (2023, methodology): “We evaluate on the oﬃcial dataset split of VQA-RAD [16], SLAKE [19], where SLAKE is composed of 642 images and 14,028 questions and VQA-RAD contains 315 images and 3,515 questions.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, positive): “Following, LLaVA-Med was evaluated on three biomedical medical visual question answering (VQA) datasets, VQA-RAD [240], SLAKE [241], and PathVQA [242].”
PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? (2023, methodology): “Vision Transformers indeed are capable of accounting for such features (Yu et al., 2021), and hence perform better on the SLAKE dataset.”