IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

By Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, Jungong Han. 2020

395 citations recorded by the source corpus.

Enabling bi-directional retrieval of images and texts is important for understanding the correspondence between vision and language. Existing methods leverage the attention mechanism to explore such correspondence in a fine-grained manner. However, most of them consider all semantics equally and thus align them uniformly, regardless of their diverse complexities. In fact, semantics are diverse (i.e. involving different kinds of semantic concepts), and humans usually follow a latent structure to combine them into understandable languages. It may be difficult to optimally capture such sophisticated correspondences in existing methods. In this paper, to address such a deficiency, we propose an Iterative Matching with Recurrent Attention Memory (IMRAM) method, in which correspondences between images and texts are captured with multiple steps of alignments. Specifically, we introduce an iterative matching scheme to explore such fine-grained correspondence progressively. A memory distillation unit is used to refine alignment knowledge from early steps to later ones. Experiment results on three benchmark datasets, i.e. Flickr8K, Flickr30K, and MS COCO, show that our IMRAM achieves state-of-the-art performance, well demonstrating its effectiveness. Experiments on a practical business advertisement dataset, named KWAI-AD, further validates the applicability of our method in practical scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Meng Liu
2. Liqiang Nie
3. Huchuan Lu
4. Jiacheng Chen
5. Leigang Qu
6. Haiwen Diao
7. Lingyun Ma
8. Yuning Jiang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Huchuan Lu
4. Xiaogang Wang
5. Dacheng Tao
6. Yi Yang
7. Hexiang Hu
8. Bernt Schiele

In Information Retrieval & Search

1. Liqiang Nie
2. Yuning Jiang
3. Hexiang Hu

In Multimodal Learning

1. Liqiang Nie
2. Meng Liu
3. Huchuan Lu
4. Jiacheng Chen
5. Leigang Qu
6. Haiwen Diao
7. Lingyun Ma
8. Yuning Jiang

Citations in the corpus

• Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “We follow [3, 46] to use RSUM, which is defined as the sum of recall metrics at K = {1, 5, 10} of both I→T (I2T) and T→I (T2I) retrievals, as a summarizing metric to gauge retrieval model’s overall performances.”
• Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, neutral): “Full-IMRAM [4] 61.”
• VLP: A Survey on Vision-language Pre-training (2022, neutral): “Vision-Language Retrieval (VLR) [81, 82, 83, 84].”
• Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “• Regarding cross-modal interaction pattern based approaches, IMRAM [3] outperforms SCAN* [15] on all criteria of two datasets.”
• Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “…alignments between image regions and sentence fragments, which produce impressive retrieval re-sults and inspire a surge of works (Wang et al. 2019c; Hu et al. 2019; Zhang et al. 2020; Chen et al. 2020; Wehrmann, Kolling, and Barros 2020) to explore more accurate fine-grained correspondence.”
• GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition (2021, background): “While the idea of learning local representations has been explored in several other contexts for natural images [7, 27, 25, 4], including image-text retrieval and text-toimage generation, these works typically require pre-trained object detection models to extract localized image features, which are”