Self-Supervised Learning from Web Data for Multimodal Retrieval

By Raul Gomez, Lluís Gómez, Jaume Gibert, Dimosthenis Karatzas. 2019

27 citations recorded by the source corpus.

Abstract Self-supervised learning from multimodal image and text data allows deep neural networks to learn powerful features with no need of human-annotated data. Web and social media platforms provide a virtually unlimited amount of this multimodal data. In this work we propose to exploit this free available data to learn a multimodal image and text embedding, aiming to leverage the semantic knowledge learned in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the proposed pipeline can learn from images with associated text without supervision and analyze the semantic structure of the learned joint image and text embedding space. We perform a thorough analysis and performance comparison of five different state-of-the-art text embeddings in three different benchmarks. We show that the embeddings learned with web and social media data have competitive performances over supervised methods in the text-based image retrieval task, and we clearly outperform the state of the art in the MIRFlickr dataset when training in the target data. Further, we demonstrate how semantic multimodal image retrieval can be performed using the learned embeddings, going beyond classical instance-level retrieval problems. Finally, we present a new dataset, InstaCities1M, composed of Instagram images and their associated texts, which can be used for fair comparison of image–text embeddings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “Vision and language (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as crossmodal retrieval [20,56], visual question answering [24], captioning [51] or visual grounding [10,29].”
Joint Learning of Localized Representations from Medical Images and Reports (2021, background): “Early works on combining image and text modalities did not focus on pre-training for downstream tasks but instead on learning aligned representations for cross-modal retrieval[80,24], on encoder-decoder tasks like image captioning[79,39,17], and on joint prediction tasks like visual question answeri”