Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction
By Hyeonwoo Noh, Paul Hongsuck Seo, Bohyung Han. 2015
337 citations recorded by the source corpus.
We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions. For the adaptive parameter prediction, we employ a separate parameter prediction network, which consists of gated recurrent unit (GRU) taking a question as its input and a fully-connected layer generating a set of candidate weights as its output. However, it is challenging to construct a parameter prediction network for a large number of parameters in the fully-connected dynamic parameter layer of the CNN. We reduce the complexity of this problem by incorporating a hashing technique, where the candidate weights given by the parameter prediction network are selected using a predefined hash function to determine individual weights in the dynamic parameter layer. The proposed network-joint network with the CNN for ImageQA and the parameter prediction network-is trained end-to-end through back-propagation, where its weights are initialized using a pre-trained CNN and GRU. The proposed algorithm illustrates the state-of-the-art performance on all available public ImageQA benchmarks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Trevor Darrell
- Jianfeng Gao
- Dacheng Tao
- Yoshua Bengio
- Percy Liang
- Sergey Levine
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Chunhua Shen
In Multimodal Learning
- Anton van den Hengel
- Christopher Kanan
- Kushal Kafle
- Qi Wu
- Marcus Rohrbach
- Trevor Darrell
- Devi Parikh
- Damien Teney
In Question Answering & Reading Comprehension
- Anton van den Hengel
- Christopher Kanan
- Kushal Kafle
- Damien Teney
- Qi Wu
- Chunhua Shen
- Peng Wang
- Anthony R. Dick
Citations in the corpus
- Learning feed-forward one-shot learners (2016, methodology): “[15] recently employed parameter prediction to induce the weights of an image classifier from text for the problem of visual question answering.”
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “The MCB is followed by an element-wise signed square-root and L2 normalization.”
- Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, background): “Recent years have seen a proliferation of datasets [19, 4] and approaches, including models based on differentiable memory [25, 24], dynamic prediction of question-specific computations [20, 2], and core improvements to the implementation of the multi-modal representation and attention mechanism [9,”
- Learning to Compose Neural Networks for Question Answering (2016, methodology): “…a highly effective visual bag-of-words model (Zhou et al., 2015), a model with dynamic network parameter prediction (but fixed network structure) (Noh et al., 2015), a more conventional attentional model (Yang et al., 2015), and a previous approach using neural module networks with no structure…”
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “• DPPnet [27]: uses the Gated Recurrent Unit (GRU) representation of question to predict certain parameters for a CNN classification network.”
- Simple Baseline for Visual Question Answering (2015, methodology): “Open-Ended Overall yes/no number others LSTMIMG [2] 54.06 - - - NMN+LSTM [1] 55.10 - - - ACK [16] 55”
- Dynamic Filter Networks (2016, methodology): “In the context of visual question answering, Noh et al. [15] introduce a dynamic parameter layer whose output is used as parameters of a fully connected layer.”
- Stacking with Auxiliary Features for Visual Question Answering (2018, background): “(Noh et al., 2016) and the Neural Module Networks (NMNs) (Andreas et al.”
- Dual Attention Networks for Multimodal Reasoning and Matching (2016, background): “Recently, dramatic advances in deep learning have broken the boundaries between vision and language, drawing growing interest in their intersection, such as visual question answering (VQA) [3, 37, 23, 35], image captioning [33, 2], image-text matching [8, 11, 20, 30], visual grounding [24, 9], etc.”
- Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, methodology): “Dynamic Parameter Prediction [30] and Question-guided Hybrid Convolution [8] utilized dynamically predicted parameters for feature fusion.”
- MarioQA: Answering Questions by Watching Gameplay Videos (2016, background): “On the other hand, image question answering (ImageQA) [3, 4, 5, 6, 10, 14, 22, 23, 24, 25, 28] addresses a holistic image understanding problem, and handles diverse recognition tasks in a single framework.”
- Learning Multi-domain Convolutional Neural Networks for Visual Tracking (2015, background): “CNNs have demonstrated their outstanding representation power in a wide range of computer vision applications [12, 28, 31, 34, 37, 41, 40, 43].”
- An Analysis of Visual Question Answering Algorithms (2017, methodology): “We only use Visual Genome to make color and positional reasoning questions.”
- FVQA: Fact-Based Visual Question Answering (2016, background): “[20, 21] formulated the VQA as a classification problem and restrict the answer only can be drawn from a fixed answer space.”
- Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, neutral): “0 DPPnet [23] 57.”
- Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”