Contrastive Learning for Weakly Supervised Phrase Grounding

By Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, Derek Hoiem. 2020

161 citations recorded by the source corpus.

Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a $\sim10\%$ absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of $5.7\%$ to achieve $76.7\%$ accuracy on Flickr30K Entities benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Among existing studies, Ilharco et al. (2021) and Gupta et al. (2020) explored cross-modality contrastive objectives related to ours, but for the purpose of probing visual-linguistic models and learning phrase grounding, respectively.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “To scale up learning, weakly-supervised visual grounding has been introduced where the bounding box and text correspondence is not available during training [11, 28, 41, 42, 78, 85].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “Following [22, 24, 82], we employ a grounding loss to supervise the prediction of the masks’ category labels.”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “Learning visual knowledge from image-text pairs has been a long-standing problem [50, 8, 14, 7, 22, 54, 58, 16, 61, 37, 11, 35], with recent success on learning deep models for visual representation [7, 22, 37, 11, 35], and for recognizing and detecting individual visual concepts (e.”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “These models are then usually optimized with objectives such as CCA [19], max margin triplet loss [15, 71, 72, 74, 75], contrastive loss [23] and, more related to our work, by maximizing text loglikelihoods conditioned on the image [14].”
Open-Vocabulary Image Segmentation (2021, methodology): “Visual grounding: Image captioning and image-text coreference datasets [9, 26, 35] enable research on the interplay of captions and grounded visual concepts [13,14,18,24,39].”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “The research community has studied visual grounding through tasks including phase grounding, object retrieval and localization, language-driven instance segmentation, and others [64,72,62,70,57,82,26,22].”
Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “manner [3, 11, 15, 18], and sometimes leverages contrastive learning techniques [3, 18].”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “Inspired from previous work on contrastive learning for weakly supervised phrase grounding [18], we define a new compatibility function for contrastive loss to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “Visual grounding: Image captioning and image text datasets [37,9,27] enable research on the interplay of captions and grounded visual concepts [14,41,15,19,25].”
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing (2022, background): “[26] Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem.”