Position Focused Attention Network for Image-Text Matching

By Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, Xin Fan. 2019

191 citations recorded by the source corpus.

Image-text matching tasks have recently attracted a lot of attention in the computer vision field. The key point of this cross-domain problem is how to accurately measure the similarity between the visual and the textual contents, which demands a fine understanding of both modalities. In this paper, we propose a novel position focused attention network (PFAN) to investigate the relation between the visual and the textual views. In this work, we integrate the object position clue to enhance the visual-text joint-embedding learning. We first split the images into blocks, by which we infer the relative position of region in the image. Then, an attention mechanism is proposed to model the relations between the image region and blocks and generate the valuable position feature, which will be further utilized to enhance the region expression and model a more reliable relationship between the visual image and the textual sentence. Experiments on the popular datasets Flickr30K and MS-COCO show the effectiveness of the proposed method. Besides the public datasets, we also conduct experiments on our collected practical news dataset (Tencent-News) to validate the practical application value of proposed method. As far as we know, this is the first attempt to test the performance on the practical application. Our method can achieve the state-of-art performance on all of these three datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, neutral): “@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 1k Test set SCAN[29] 48.6 77.7 85.2 67.4 90.3 95.8 58.8 88.4 94.8 72.7 94.8 98.4 SCG[30] 49.3 76.4 85.6 71.8 90.8 94.8 61.4 88.9 95.1 76.6 96.3 99.2 PFAN[31] 50.4 78.7 86.1 70.0 91.8 95.0 61.6 89.6 95.2 76.5 96.3 99.0 ViLBERT[14] 58.2 84.9 91.5 - - - - ”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, neutral): “edding learning afterward. Pixel-BERT 11 Model TR IR R@1 R@5 R@10 R@1 R@5 R@10 VSE++[10] 52.9 80.5 87.2 39.6 70.1 79.5 SCAN[17] 67.4 90.3 95.8 48.6 77.7 85.2 SCG[26] 71.8 90.8 94.8 49.3 76.4 85.6 PFAN[36] 70.0 91.8 95.0 50.4 78.7 86.1 ViLBERT[21] - - - 58.2 84.9 91.5 Unicoder-VL[18] 86.2 96.3 99.0 7”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, background): “Video-Text Retrieval has received wide attention with the exploitation of the huge multimedia data and rich application scenarios.”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “…visual and textual observations, a large proportion of methods (Liu et al. 2017; Nam, Ha, and Kim 2017; Lee et al. 2018; Song and Soleymani 2019; Wang et al. 2019c; Li et al. 2019; Wang et al. 2020) (Faghri et al. 2017) map the whole image and the full sentence into a common vector space, and…”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, background): “…performance in search and retrieval applications, both monolingually for English (Nam et al., 2017; Faghri et al., 2018; Zheng et al., 2020; Wang et al., 2019a; Shi et al., 2019) and in multilingual contexts (Gella et al., 2017; Kádár et al., 2018; Kim et al., 2020; Wehrmann et al., 2019;…”
Graph Structured Network for Image-Text Matching (2020, methodology): “(4) state-of-the-art methods SCAN [14], BFAN [18], PFAN [30].”
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, background): “VSE VSE++ SCAN PFAN ViLBERT VLBERT FashionBERT ImageBERT OSCAR Kaleido-BERT Tasks [33] [16] [36] [71] [60] [45] [21] [55] [42] Ours”