Knowledge Aware Semantic Concept Expansion for Image-Text Matching

By Botian Shi, Lei Ji, Pan Lu, Zhendong Niu, Nan Duan. 2019

83 citations recorded by the source corpus.

Image-text matching is a vital cross-modality task in artificial intelligence and has attracted increasing attention in recent years. Existing works have shown that learning semantic concepts is useful to enhance image representation and can significantly improve the performance of both image-to-text and text-to-image retrieval. However, existing models simply detect semantic concepts from a given image, which are less likely to deal with long-tail and occlusion concepts. Frequently co-occurred concepts in the same scene, e.g. bedroom and bed, can provide common-sense knowledge to discover other semantic-related concepts. In this paper, we develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge. Moreover, we propose a novel model to incorporate this knowledge to improve image-text matching. Specifically, semantic concepts are detected from images and then expanded by the SCG. After learning to select relevant contextual concepts, we fuse their representations with the image embedding feature to feed into the matching module. Extensive experiments are conducted on Flickr30K and MSCOCO datasets, and prove that our model achieves state-of-the-art results due to the effectiveness of incorporating the external SCG.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Lijuan Wang
4. Mohit Bansal
5. Furu Wei
6. Iryna Gurevych
7. Xiaowei Hu
8. Yejin Choi

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Dacheng Tao
3. Luc Van Gool
4. Lijuan Wang
5. Ming-Hsuan Yang
6. Zhe Gan
7. Xiaogang Wang
8. Trevor Darrell

In Multimodal Learning

1. Lijuan Wang
2. Jianfeng Gao
3. Xiaowei Hu
4. Zhe Gan
5. Pengchuan Zhang
6. Zicheng Liu
7. Jianfeng Wang
8. Chunyuan Li

Citations in the corpus

• Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “72.4 DPC [46] - 65.689.895.5 47.179.990.0 41.270.581.1 25.353.466.4 CAMP [42] - 72.394.898.3 58.587.995.0 50.182.189.7 39.068.980.2 SCAN [18] - 72.794.898.4 58.888.494.8 50.482.290.0 38.669.380.4 SCG [33] - 76.696.399.2 61.488.995.1 56.684.592.0 39.268.081.3 PFAN [41] - 76.596.399.0 61.689.695.2 -- ”
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, neutral): “4 SCG[30] 49.”
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, neutral): “2 SCG[26] 71.”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “These knowledge graphs can be used for various multi-modal tasks such as image-text matching [87], visual question answering [88], and recommendation [89].”
• Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “To improve the discriminative ability of the unified embeddings, many strategies such as semantic concept learning (Huang et al. 2018; Shi et al. 2019) and region relationship reasoning (Li et al.”
• Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, background): “Efficient approaches to cross-modal image-text retrieval relied on the induction of shared multi-modal visual-semantic embedding spaces (VSEs) (Frome et al., 2013; Faghri et al., 2018; Shi et al., 2019; Mahajan et al., 2019).”
• IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (2020, methodology): “We omit models that require additional data augmentation [19, 22, 17, 14, 2, 10].”