Contrastive Learning of Medical Visual Representations from Paired Images and Text

By Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, Curtis P. Langlotz. 2020

1,105 citations recorded by the source corpus.

Learning visual representations of medical images is core to medical image understanding but its progress has been held back by the small size of hand-labeled datasets. Existing work commonly relies on transferring weights from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. We propose an alternative unsupervised strategy to learn medical visual representations directly from the naturally occurring pairing of images and textual data. Our method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test our method by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that our method leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “Unlike the aforementioned works, another line of works focus on the way of learning visual representation from natural language supervisions, including both generative [5] and discriminative [48, 49, 58] methods.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Models like CLIP learn image classification by matching images with their captions, contrastively [130, 91, 62].”
Learning to Prompt for Vision-Language Models (2021, methodology): “…language models, formally deﬁned by Petroni et al. (2019) as “ﬁll-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
MERLOT: Multimodal Neural Script Knowledge Models (2021, neutral): “Contrastive frame-transcript matching [126, 89].”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “(ii) Another line of works focuses on learning transferable visual representation from natural language supervisions, including generative [12, 48] and contrastive methods [16,29,47,59,60,74].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “This training scheme has been successfully employed in different fields such as image and video classification [25], [53], [54], as well as natural language processing [55], [56] and audio representation learning [57], [58], [59].”
Conditional Prompt Learning for Vision-Language Models (2022, background): “Vision-Language Models We mainly review studies focused on aligning images and texts to learn a joint embedding space [24, 40, 59].”
Poisoning and Backdooring Contrastive Learning (2021, background): “Instead of operating on objects of just one type, multimodal contrastive learning uses multiple domains simultaneously (e.g., images and text) (Zhang et al., 2020).”
Robust fine-tuning of zero-shot models (2021, background): “One increasingly popular class of vision models are those pre-trained with auxiliary language supervision, which can be used for zero-shot inference [18, 44, 75, 79, 84, 107, 109].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “With large-scale image-text paired data available on the Internet, representation learning with text supervision [14,19,30,32, 37, 39, 49, 59, 89] has been shown to be successful in transferring to various down stream tasks such as visual question answering [2, 92] and visual reasoning [87].”
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology (2022, background): “The leaps made by AI in connecting images and text 107,140 are of particular relevance to histopathology, in which algorithms may not only learn from but also learn to generate basic histopathology reports.”