Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?

By Sedigheh Eslami, Gerard de Melo, Christoph Meinel. 2021

159 citations recorded by the source corpus.

Contrastive Language--Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image--text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. This work evaluates the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). To this end, we present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments are conducted on two MedVQA benchmark datasets and investigate two MedVQA methods, MEVF (Mixture of Enhanced Visual Features) and QCR (Question answering via Conditional Reasoning). For each of these, we assess the merits of visual representation learning using PubMedCLIP, the original CLIP, and state-of-the-art MAML (Model-Agnostic Meta-Learning) networks pre-trained only on visual data. We open source the code for our MedVQA pipeline and pre-training PubMedCLIP. CLIP and PubMedCLIP achieve improvements in comparison to MAML's visual encoder. PubMedCLIP achieves the best results with gains in the overall accuracy of up to 3%. Individual examples illustrate the strengths of PubMedCLIP in comparison to the previously widely used MAML networks. Visual representation learning with language supervision in PubMedCLIP leads to noticeable improvements for MedVQA. Our experiments reveal distributional differences in the two MedVQA benchmark datasets that have not been imparted in previous work and cause different back-end visual encoders in PubMedCLIP to exhibit different behavior on these datasets. Moreover, we witness fundamental performance differences of VQA in general versus medical domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “Since the release of our work, similar image-text pretraining strategies have been used to improve more downstream healthcare tasks including image regeneration (Wang et al., 2021), medical visual question answering (Eslami et al., 2021) and clinical risk prediction (Zang and Wang, 2021), etc.”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “To further examine the answers provided by BiomedCLIP, we evaluate BiomedCLIP on some of the most challenging examples highlighted in the PubMedCLIP paper [25], where all prior state-of-the-art models, including Pub-MedCLIP, failed to answer correctly ( Fig.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, methodology): “As an initial experiment, we perform continuous pretraining of the CLIP model (ResNet-50 and ViT-Base-32) on the union of three existing retrieval datasets RSITMD, RSICD, and UCM ( RET-3 ) 1 We used the standard CLIP training setup, following common practice [62], [63], [64].”
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, background): “[14], have recently been applied to multiple vision tasks [57, 74, 6, 53], but rarely to the medical domain [16, 75].”