Dual-path Convolutional Image-Text Embeddings with Instance Loss

By Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, Yi-Dong Shen. 2017

595 citations recorded by the source corpus.

Matching images and sentences demands a fine understanding of both modalities. In this article, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image/text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss on heterogeneous features (i.e., text and image features) is less effective, because it is hard to find appropriate triplets at the beginning. So the naive way of using the ranking loss may compromise the network from learning inter-modal relationship. To address this problem, we propose the instance loss, which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image/text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this article constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language-based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Position Focused Attention Network for Image-Text Matching (2019, neutral): “…al. , 2017] 49.8 67.5 - 36.0 55.6 - DAN [Nam et al. , 2017] 55.0 81.8 89.0 39.4 69.2 79.1 VSE++ [Faghri et al., 2018] 52.9 - 87.2 39.6 - 79.5 DPC [Zheng et al ., 2018] 55.6 81.9 89.5 39.1 69.2 80.9 SCO [Huang et al., 2018] 55 Image-to-Text Retrieval Text-to-Image Retrieval R@1 R@5 R@10 R@1 R@5…”
Graph Structured Network for Image-Text Matching (2020, background): “6 DPC [35] ft ResNet-152 ft ResNet-152 65.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, background): “[42] proposed to discriminatively embed images and texts into a shared semantic space with an instance loss.”
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (2020, methodology): “Image Retrieval R@sum R@1 R@5 R@10 R@1 R@5 R@10 1K DPC [28] 65.6 89.8 95.5 47.1 79.9 90.0 467.9 SCO [7] 69.9 92.9 97.”
Multi-Modality Cross Attention Network for Image and Sentence Matching (2020, background): “The goal of such a cross-modal matching task is how to accurately measure the visual-semantic similarity between an image and a sentence, and is related to many visionlanguage tasks including image-sentence cross-modal retrieval [21, 48, 55], visual captioning [1, 5], visual grounding [8, 53] and vi”
University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization (2020, methodology): “In light of the recent development in image-language bi-directional retrieval, we adopt one classification loss called instance loss [36] to train the baseline.”
Each Part Matters: Local Patterns Facilitate Cross-View Geo-Localization (2020, background): “They apply the instance loss [25], [26] to optimize the network and has achieved a competitive result.”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, background): “Multilingual zero-shot results, where we ﬁne-tune on the English Multi30k captions and test on the captions in other languages, are shown in Table 7.”
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval (2023, methodology): “To achieve this goal, the commonly utilized ID loss [54] is also adopted along with SDM loss and IRR loss to optimize IRRA.”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “score between the input image and text pair [114], [155], [156], [157], [158].”