Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

By Jiuxiang Gu, Jianfei Cai, Shafiq R. Joty, Li Niu, G. Wang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

385 citations recorded by the source corpus.

Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/724b253a55e86ad230ba05c7eb78f249e09258d9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Multimodal Representation Learning: A Survey (2019, background): “[86] proposed to learn cross-modal embeddings used for retrieval.”
Graph Optimal Transport for Cross-Domain Alignment (2020, neutral): “9 GXN (ResNet) (Gu et al., 2018) 42.”
Visual Semantic Reasoning for Image-Text Matching (2019, methodology): “We follow the splits of [5, 8, 15, 23] for MSCOCO, which contains 113,287 images for training, 5,000 images for validation and 5000 images for testing.”
Cross-Modality Person Re-Identification With Shared-Specific Feature Transfer (2020, background): “Cross-modality retrieval has been widely studied for heterogeneous face recognition [15] and text-to-image retrieval [9, 16, 21, 22, 27].”
Deep Supervised Cross-Modal Retrieval (2019, background): “Inspired by the great success of deep neural networks in representation learning [14], a large number of deep learning-based approaches [2, 33, 19, 36, 21, 25, 7] have been proposed to learn a common presentation space for multimedia data.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, neutral): “R@1 R@5 R@10 R@1 R@5 R@10 AR R@1 R@5 R@10 R@1 R@5 R@10 AR proposed method with both task-speciﬁc models: VSE++ (Faghri et al., 2017), GXN (Gu et al., 2018), SCO (Huang et al., 2018), SCAN (Lee et al., 2018), R-SCAN (Lee et al., 2019), CAMP (Wang et al., 2019) and CAAN (Zhang et al., 2020), and V+L p”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, background): “Our re-ported results have been recently outperformed by Gu et al. (2018); Huang et al. (2018b); Lee et al. (2018), which are more sophisticated methods that incorporate generative modelling, reinforcement learning and attention.”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “Similarity Prediction Most existing works (Faghri et al. 2017; Wang, Li, and Lazebnik 2016; Zheng et al. 2017; Vendrov et al. 2016; Gu et al. 2018) for image-text matching learned the joint embedding and the similarity measures for cross-modal matching.”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, result): “Compared with recent works, VSE++ [10], GXN [17] and CNP [29], our result is also competitive.”
Multi-Modal Open-Domain Dialogue (2020, background): “A variety of sequence-to-sequence (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018) and retrieval-based (Gu et al., 2018; Faghri et al., 2018; Nam et al., 2016) models have been applied to these tasks, however they do not go beyond the one-turn text generation expected for captioning…”
Graph Structured Network for Image-Text Matching (2020, background): “The main goal of global correspondence learning methods [19, 4, 29, 29, 31, 22, 34, 1] is to maximize similarity of matched image-text pairs.”
Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing (2020, methodology): “Compared with the supervised counterparts, unsupervised cross-modal hashing methods [8, 45, 13, 36] only relied on the correlation information from the paired data, making it easier to be deployed to other scenarios.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “[9] further incorporate image and caption generation in multi-task framework.”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Great strides have been made towards cross-modal image retrieval [12, 28], predominantly using text as a query [20, 45].”
Position Focused Attention Network for Image-Text Matching (2019, methodology): “A popular framework to model the relationship between image and text is the two-branch embedding network, one branch projects the image and another models the text, the shared subspace is learned by the popular triplet loss [Gu et al., 2018; Wang et al., 2018; Nam et al., 2017].”