Learning to Answer Questions from Image Using Convolutional Neural Network

By Lin Ma, Zhengdong Lu, Hang Li. 2015

268 citations recorded by the source corpus.

In this paper, we propose to employ the convolutional neural network (CNN) for the image question answering (QA) task. Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: one image CNN to encode the image content, one sentence CNN to compose the words of the question, and one multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on the DAQUAR and COCO-QA datasets, which are two benchmark datasets for image QA, with the performances significantly outperforming the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016, background): “Models may train recurrent networks for answer generation [10, 28, 41], multiclass classifiers over common answers [4, 24, 25, 32, 48, 49], or binary classifiers on imagequestion-answer triples [9, 16, 33].”
Neural Module Networks (2015, background): “Several neural models for visual questioning have already been proposed in the literature [33, 26, 10], all of which use standard deep sequence modeling machinery to construct a joint embedding of image and text, which is immediately mapped to a distribution over answers.”
Stacked Attention Networks for Image Question Answering (2015, methodology): “(13) The max-pooling over these vectors is a coordinate-wise max operation.”
Visual7W: Grounded Question Answering in Images (2015, background): “The proposed models range from probabilistic inference [27, 44, 51] and recurrent neural networks [1, 7, 28, 36] to convolutional networks [26].”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, methodology): “We compare with 2-VIS+BLSTM [14], IMG-CNN [12] and SAN [23] on COCO-QA.”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, background): “The two main types of soft attention mechanisms differ in the function that aligns the input feature vector and the candidate feature vectors in order to compute the soft attention weights.”
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images (2015, background): “We like to note that shortly after this work, several neural-based models [24, 19, 7] have also been suggested.”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “Following the previously deﬁned experimental method, we exclude multiple word answers (Mali-nowski et al., 2015; Ma et al., 2015).”
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, methodology): “[17] encode both images and questions with a CNN.”
Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction (2015, methodology): “ImageQA problem has a short history in computer vision and machine learning community, but there already exist several approaches [10, 16, 17, 18, 23].”
What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (2015, methodology): “[32] used CNNs to both extract image features and sentence features, and fuse the features together with a multi-modal CNN. Antol et al .”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “In the following subsections, we will survey work that follows these two approaches.”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, methodology): “Convolutional Neural Network (CNN) that models language (Kim, 2014; Kalchbrenner et al, 2014; Ma et al, 2016; Yang et al, 2015) is gaining popularity due to its speed and good accuracy for the language-oriented tasks.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, background): “Using a variety of datasets [11, 2, 8, 10, 3, 5], models based on multi-modal representation and attention [18–25], deep network architectures [26, 12, 27, 28], and dynamic memory nets [29] have been developed.”
Knowledge Acquisition for Visual Question Answering via Iterative Querying (2017, background): “Existing VQA models vary from symbolic approaches [25, 38], neural-based approaches [9, 23, 24, 26, 31], to a hybrid scheme of the former two [2].”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, unknown): “ing [25, 9], and so on.”
Leveraging Visual Question Answering for Image-Caption Ranking (2016, methodology): “Recent VQA models [9,6,8,1,4] explore state-of-the-art deep learning techniques combining Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).”