Leveraging Visual Question Answering for Image-Caption Ranking

By Xiaoyu Lin, Devi Parikh. 2016

88 citations recorded by the source corpus.

Visual Question Answering (VQA) is the task of taking as input an image and a free-form natural language question about the image, and producing an accurate answer. In this work we view VQA as a “feature extraction” module to extract image and caption representations. We employ these representations for the task of image-caption ranking. Each feature dimension captures (imagines) whether a fact (question-answer pair) could plausibly be true for the image and caption. This allows the model to interpret images and captions from a wide variety of perspectives. We propose score-level and representation-level fusion models to incorporate VQA knowledge in an existing state-of-the-art VQA-agnostic image-caption ranking model. We find that incorporating and reasoning about consistency between images and captions significantly improves performance. Concretely, our model improves state-of-the-art on caption retrieval by 7.1 % and on image retrieval by 4.4 % on the MSCOCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

From Recognition to Cognition: Visual Commonsense Reasoning (2018, methodology): “We note that BottomUpTopDown, MLB, and MUTAN all treat VQA as a multilabel classification over the top 1000 answers [4, 50].”
Visual Semantic Reasoning for Image-Text Matching (2019, neutral): “…Methods Caption Retrieval Image Retrieval R@ 1 R@ 5 R@ 10 R@ 1 R@ 5 R@ 10 (R-CNN, AlexNet) DVSA CVPR (cid:48) 15 [15] 38 [20] 39.4 67.9 80.9 25.1 59.8 76.6 OEM ICLR (cid:48) 16 [35] 46.7 - 88.9 37.9 - 85.9 VQA ECCV (cid:48) 16 [29] 50.5 80.1 89.7 37.0 70.9 82.9 SMlstm CVPR (cid:48) 17 [13] Table 1.”
Active Learning for Visual Question Answering: An Empirical Study (2017, background): “In fact, VQA can play several roles from answering any generic question about an image, to answering only speciﬁc question types ( e . g ., questions with “yes/no” answers, or counting questions), to being a submodule in some other task ( e . g ., image captioning as in [19]).”
Watch What You Just Said: Image Captioning with Text-Conditional Attention (2016, background): “Various methods have shown to improve performance with regionlevel information [8, 15], external knowledge [1], and even from question-answering [24].”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, methodology): “some point in the pipeline. Concerning image processing, the standard approach consists in using Convolutional Neural Networks (CNNs), usually pretrained on image classification tasks. In particular, [9, 15, 20, 29, 44] used VGGs, [10, 11, 16, 31] used ResNets. The problem with these kinds of CNNs i”
Learning Semantic Concepts and Order for Image and Sentence Matching (2017, unknown): “6 VQA [20] 33.”
Semantic bottleneck for computer vision tasks (2018, background): “ottleneck 3 approach, both the image (CNN features) and the question are fed into the LSTM to answer a question about an image. Once the question is encoded, the answers can be generated by the LSTM. [16] study the problem of using VQA knowledge to improve image-caption ranking. [17], motivated by t”
Image Caption Generation with Text-Conditional Semantic Attention (2016, methodology): “or both image-text embedding and sentence generating. Various methods have shown to improve performance with region-level information [8, 16], external knowledge [1], and even from question-answering [24]. Our method differs from them by considering attention from textual context in caption generati”
Deep Cross-Modal Projection Learning for Image-Text Matching (2018, background): “Exploring the relationship between image and natural language has recently attracted great interest among researchers, due to its great importance in various applications, such as bi-directional image and text retrieval [44, 22], natural language object retrieval [10], image captioning [43, 35], and”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, unknown): “2 14 VQA-A [18] fixed VGG-19 ft RNN 23.”
Image Pivoting for Learning Multilingual Multimodal Representations (2017, unknown): “Recently Lin and Parikh (2016) have leveraged visual question answering models to encode images and descriptions into the same space.”