MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

By Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, Jimeng Sun. Conference on Empirical Methods in Natural Language Processing, 2022

1,038 citations recorded by the source corpus.

Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning, thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using 200K data). The code is available at https://github.com/RyanWangZf/MedCLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/cdd9c1d23f9e89d5113f3e31821bb174c6a6afed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Matthew P. Lungren

Citations in the corpus

CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, neutral): “, CLIP [14], have recently been applied to multiple vision tasks [5,44,47,63], but rarely to the medical domain [16, 64].”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, methodology): “As an initial experiment, we perform continuous pretraining of the CLIP model (ResNet-50 and ViT-Base-32) on the union of three existing retrieval datasets RSITMD, RSICD, and UCM ( RET-3 ) 1 We used the standard CLIP training setup, following common practice [62], [63], [64].”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “This contrasts with PubMedCLIP [25] and MedCLIP [26], where PubMedCLIP simply fine-tunes the original CLIP on a limited dataset from PubMed Central and MedCLIP incorporates medical knowledge into the learning process but still within the constraints of the original CLIP architecture.”