Heterogeneous Attention Network for Effective and Efficient Cross-modal Retrieval

By Tan Yu, Yi Yang, Yi Li, Lin Liu, Hongliang Fei, Ping Li. 2021

56 citations recorded by the source corpus.

Traditionally, the task of cross-modal retrieval is tackled through joint embedding. However, the global matching used in joint embedding methods often fails to effectively describe matchings between local regions of the image and words in the text. Hence they may not be effective in capturing the relevance between the text and the image. In this work, we propose a heterogeneous attention network (HAN) for effective and efficient cross-modal retrieval. The proposed HAN represents an image by a set of bounding box features and a sentence by a set of word features. The relevance between the image and the sentence is determined by the set-to-set matching between the set of word features and the set of bounding box features. To enhance the matching effectiveness, we exploit the proposed heterogeneous attention layer to provide the cross-modal context for word features as well as bounding box features. Meanwhile, to optimize the metric more effectively, we propose a new soft-max triplet loss, which adaptively gives more attention to harder negatives and thus trains the proposed HAN in a more effective manner compared with the original triplet loss. Meanwhile, the proposed HAN is efficient, and its lightweight architecture only needs a single GPU card for training. Extensive experiments conducted on two public benchmarks demonstrate the effectiveness and efficiency of our HAN. This work has been deployed in production Baidu Search Ads and is part of the "PaddleBox'' platform.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Wei Wang
6. Fei Huang
7. Luo Si
8. Haiyang Xu

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Fei Huang
6. Wei Wang
7. Haiyang Xu
8. Luo Si

In Information Retrieval & Search

1. Ming Yan
2. Songfang Huang
3. Wei Wang

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Haiyang Xu
4. Ming Yan
5. Ji Zhang
6. Weihua Chen
7. Fan Wang
8. Songfang Huang

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, background): “Visual question answering involves the cross-modal fusion and alignment between image and text, which is related to the topic of cross-modal retrieval [12, 26, 28, 32, 47, 60, 61, 84, 92].”