Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval

By Niluthpol Chowdhury Mithun, Rameswar Panda, Evangelos E. Papalexakis, Amit K. Roy-Chowdhury. 2018

81 citations recorded by the source corpus.

Cross-modal retrieval between visual data and natural language description remains a long-standing challenge in multimedia. While recent image-text retrieval methods offer great promise by learning deep representations aligned across modalities, most of these methods are plagued by the issue of training with small-scale datasets covering a limited number of images with ground-truth sentences. Moreover, it is extremely expensive to create a larger dataset by annotating millions of images with sentences and may lead to a biased model. Inspired by the recent success of webly supervised learning in deep neural networks, we capitalize on readily-available web images with noisy annotations to learn robust image-text joint representation. Specifically, our main idea is to leverage web images and corresponding tags, along with fully annotated datasets, in training for learning the visual-semantic joint embedding. We propose a two-stage approach for the task that can augment a typical supervised pair-wise ranking loss based formulation with weakly-annotated web images to learn a more robust visual-semantic embedding. Experiments on two standard benchmark datasets demonstrate that our method achieves a significant performance gain in image-text retrieval compared to state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “In the deep learning era, Mithun et al. (2018) demonstrated an additional set of (image, text) pairs collected from the internet could improve retrieval performance and several new automatically constructed datasets such as Conceptual Captions (Sharma et al., 2018), LAIT (Qi et al., 2020), and…”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Another approach is to use text captions, and label each image with key words present in the caption [50, 30, 39, 48].”
Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [47, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [16, 18, 28, 29, 40, 52, 56, 62, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “Despite lacking a multimodal attention mechanism, this approach has been popular in image and video domains because of its simplicity and effectiveness for retrieval applications (e.g., Gong et al., 2014; Wang et al., 2016; Chowdhury et al., 2018; Miech et al., 2018).”
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (2020, methodology): “To compare our proposed method with the state-of-the-art methods, we adopt the same evaluation metrics in all datasets as [17, 13, 5].”
Graph Structured Network for Image-Text Matching (2020, background): “The main goal of global correspondence learning methods [19, 4, 29, 29, 31, 22, 34, 1] is to maximize similarity of matched image-text pairs.”