Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

By Qi Wu, Chunhua Shen, Peng Wang, Anthony R. Dick, Anton van den Hengel. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016

383 citations recorded by the source corpus.

Much of the recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We further show that the same mechanism can be used to incorporate external knowledge, which is critically important for answering high level visual questions. Specifically, we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. It particularly allows questions to be asked where the image alone does not contain the information required to select the appropriate answer. Our final model achieves the best reported results for both image captioning and visual question answering on several of the major benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6acd385b2742f65359efb99543ebfb9a0d1b850f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “, 2018) ResNet-152 LSTM CorefNMN 7 (Wu et al., 2018) VGG LSTM CoAtt-GAN ADAM 3 (Niu et al.”
PaperRobot: Incremental Draft Generation of Scientific Ideas (2019, background): “…(Lebret et al., 2016; Chisholm et al., 2017; Liu et al., 2018; Sha et al., 2018; Kaffee et al., 2018; Wang et al., 2018a; Wiseman et al., 2018), and image/video captions based on background entities and events (Krishnamoorthy et al., 2013; Wu et al., 2018; Whitehead et al., 2018; Lu et al., 2018).”
Graph R-CNN for Scene Graph Generation (2018, background): “These scene graphs form an interpretable structured representation of the image that can support higher-level visual intelligence tasks such as captioning [24,39], visual question answering [1,11,35,37–39], and image-grounded dialog [3].”
Memory Grounded Conversational Reasoning (2019, background): “Visual QA systems answer queries based on the contexts from provided images (Antol et al., 2015; Wang et al., 2018; Wu et al., 2018).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Image captioning methods such as [22, 65, 142, 144, 150] use this dataset for their experiments.”
CNN-RNN Based Intelligent Recommendation for Online Medical Pre-Diagnosis Support (2020, methodology): “[40] developed an attribute-based visual question answering model according to the CNNRNN architecture.”
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (2021, background): “Descriptive representations such as attributes can often be extracted from this data, and form a strong basis for generating stronger image captions [71, 66, 70] and more effective image retrieval [25, 5, 51, 34].”