Dynamic Modality Interaction Modeling for Image-Text Retrieval

By Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, Liqiang Nie. 2021

225 citations recorded by the source corpus.

Image-text retrieval is a fundamental and crucial branch in information retrieval. Although much progress has been made in bridging vision and language, it remains challenging because of the difficult intra-modal reasoning and cross-modal alignment. Existing modality interaction methods have achieved impressive results on public datasets. However, they heavily rely on expert experience and empirical feedback towards the design of interaction patterns, therefore, lacking flexibility. To address these issues, we develop a novel modality interaction modeling network based upon the routing mechanism, which is the first unified and dynamic multimodal interaction framework towards image-text retrieval. In particular, we first design four types of cells as basic units to explore different levels of modality interactions, and then connect them in a dense strategy to construct a routing space. To endow the model with the capability of path decision, we integrate a dynamic router in each cell for pattern exploration. As the routers are conditioned on inputs, our model can dynamically learn different activated paths for different data. Extensive experiments on two benchmark datasets, i.e., Flickr30K and MS-COCO, verify the superiority of our model compared with several state-of-the-art baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Wei Wang
6. Fei Huang
7. Luo Si
8. Haiyang Xu

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Fei Huang
6. Wei Wang
7. Haiyang Xu
8. Luo Si

In Information Retrieval & Search

1. Ming Yan
2. Songfang Huang
3. Wei Wang

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Haiyang Xu
4. Ming Yan
5. Ji Zhang
6. Weihua Chen
7. Fan Wang
8. Songfang Huang

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, background): “Visual question answering involves the cross-modal fusion and alignment between image and text, which is related to the topic of cross-modal retrieval [12, 26, 28, 32, 47, 60, 61, 84, 92].”