Joint Learning of Localized Representations from Medical Images and Reports

By Philip Müller, Georgios Kaissis, Cong Zou, Daniel Munich, Daniel Ruckert Technical University of Munich, I. -. London. 2021

135 citations recorded by the source corpus.

Contrastive learning has proven effective for pre-training image models on unlabeled data with promising results for tasks such as medical image classification. Using paired text (like radiological reports) during pre-training improves the results even further. Still, most existing methods target image classification downstream tasks and may not be optimal for localized tasks like semantic segmentation or object detection. We therefore propose Localized representation learning from Vision and Text (LoVT), to our best knowledge, the first text-supervised pre-training method that targets localized medical imaging tasks. Our method combines instance-level image-report contrastive learning with local contrastive learning on image region and report sentence representations. We evaluate LoVT and commonly used pre-training methods on an evaluation framework of 18 localized tasks on chest X-rays from five public datasets. LoVT performs best on 10 of the 18 studied tasks making it the preferred method of choice for localized tasks.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Le Lu
2. Matthew P. Lungren
3. Yuhao Zhang
4. Curtis P. Langlotz
5. Hoifung Poon
6. Sheng Zhang
7. Ozan Oktay
8. Daniel Coelho de Castro

In Radiology, Nuclear Medicine and Imaging

1. Matthew P. Lungren
2. Curtis P. Langlotz
3. Le Lu

In Multimodal Learning

1. Matthew P. Lungren
2. Christopher D. Manning
3. Curtis P. Langlotz
4. Yasuhide Miura
5. Yuhao Zhang
6. Le Lu
7. Serena Yeung
8. Marco Baroni

Citations in the corpus

• Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “M¨uller et al. (2021) proposed a similar method, LoVT, for localized medical imaging tasks.”
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “LoVT [34] proposes a similar pretraining approach that aligns local representations of image regions and report sentences.”
• LViT: Language Meets Vision Transformer in Medical Image Segmentation (2022, methodology): “Inspired by VLP in natural image, a few studies have started to utilize text information for assisting with medical image analysis [32], [33], [34].”
• Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, methodology): “Related self-supervised VLP work [30,31,45,56,85] has achieved impressive downstream classification and zero-shot classification performance.”