Graph Structured Network for Image-Text Matching

By Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Yongdong Zhang. Computer Vision and Pattern Recognition, 2020

305 citations recorded by the source corpus.

Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Existing works learn coarse correspondence based on object co-occurrence statistics, while failing to learn fine-grained phrase correspondence. In this paper, we present a novel Graph Structured Matching Network (GSMN) to learn fine-grained correspondence. The GSMN explicitly models object, relation and attribute as a structured phrase, which not only allows to learn correspondence of object, relation and attribute separately, but also benefits to learn fine-grained correspondence of structured phrase. This is achieved by node-level matching and structure-level matching. The node-level matching associates each node with its relevant nodes from another modality, where the node can be object, relation or attribute. The associated nodes then jointly infer fine-grained correspondence by fusing neighborhood associations at structure-level matching. Comprehensive experiments show that GSMN outperforms state-of-the-art methods on benchmarks, with relative Recall@1 improvements of nearly 7% and 2% on Flickr30K and MSCOCO, respectively. Code will be released at: https://github.com/CrossmodalGroup/GSMN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/84a65d4b4d966fae04cd95e34d12c09719be93f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, methodology): “Recent works [4, 13, 15, 28, 36, 37, 56] exploit the availability of pre-computed region-level features extracted from the Faster-RCNN [47] object detector.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “To further tackle the two aforementioned challenges, recent methods combining intra- and intermodal interactions are developed, such as the serial pattern [22] and the parallel pattern [35, 40] displayed in Figure 2(c).”