PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?
By Sedigheh Eslami, Christoph Meinel, Gerard de Melo. Findings, 2023
310 citations recorded by the source corpus.
Contrastive Language–Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image–text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. In this work, we evaluate the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). We present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments conducted on two MedVQA benchmark datasets illustrate that PubMedCLIP achieves superior results improving the overall accuracy up to 3% in comparison to the state-of-the-art Model-Agnostic Meta-Learning (MAML) networks pre-trained only on visual data. The PubMedCLIP model with different back-ends, the source code for pre-training them and reproducing our MedVQA pipeline is publicly available at https://github.com/sarahESL/PubMedCLIP.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/61c0b6a5e7aea48a1376b61a4a737137d602b242
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Chunyuan Li
- Jianfeng Gao
- Jianwei Yang
- Haotian Liu
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Trevor Darrell
Citations in the corpus
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”