Multimodal Representation Learning via Maximization of Local Mutual Information

By Ruizhi Liao, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth J. Berkowitz, Steven Horng, Polina Golland, William M. Wells. 2021

50 citations recorded by the source corpus.

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Liao et al. (2021) trained image and text encoders by encouraging the resulting representations to exhibit high local mutual information.”
Joint Learning of Localized Representations from Medical Images and Reports (2021, unknown): “Most related to our work is the recently published local-mi[44] that does contrastive learning on report sentences and image regions but aligns each sentence with its most related region instead of using an alignment model like our method.”
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “[35] learn multimodal representations by maximizing the mutual information between local features of medical images and text.”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, methodology): “Related self-supervised VLP work [29, 84, 44, 30, 55] has achieved impressive downstream classification and zero-shot classification performance.”