GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

By Liyue Shen, Matthew P. Lungren, Serena Yeung. 2021

567 citations recorded by the source corpus.

In recent years, the growing utilization of medical imaging is placing an increasing burden on radiologists. Deep learning provides a promising solution for automatic medical image analysis and clinical decision support. However, large-scale manually labeled datasets required for training deep neural networks are difficult and expensive to obtain for medical images. The purpose of this work is to develop label-efficient multimodal medical imaging representations by leveraging radiology reports. We propose an attention-based framework for learning global and local representations by contrasting image sub-regions and words in the paired report. In addition, we propose methods to leverage the learned representations for various downstream medical image recognition tasks with limited labels. Our results demonstrate high-performance and label-efficiency for image-text retrieval, classification (finetuning and zerosshot settings), and segmentation on different datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “, 2021) or GloRIA (Huang et al., 2021), mainly because such comparisons are included in these studies.”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “Despite this, the existing VLP work to date considers alignment between only single images and reports [9,32,46,81], going so far as to remove temporal content from reports in training data to prevent ‘hallucinations’ in downstream report generation [54].”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, methodology): “Related self-supervised VLP work [30,31,45,56,85] has achieved impressive downstream classification and zero-shot classification performance.”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, methodology): “GLoRIA [33] uses the same general-domain vocabulary and extends ConVIRT by jointly learning multimodal global and local representations of medical images via contrasting attention weighted image regions with words in the paired reports.”
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022, unknown): “Our method obtains better performances than the state-of-the-art GLoRIA (Huang et al., 2021) using only 10% pre-training data.”