Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction

By Hyeonwoo Noh, Paul Hongsuck Seo, Bohyung Han. 2015

337 citations recorded by the source corpus.

We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions. For the adaptive parameter prediction, we employ a separate parameter prediction network, which consists of gated recurrent unit (GRU) taking a question as its input and a fully-connected layer generating a set of candidate weights as its output. However, it is challenging to construct a parameter prediction network for a large number of parameters in the fully-connected dynamic parameter layer of the CNN. We reduce the complexity of this problem by incorporating a hashing technique, where the candidate weights given by the parameter prediction network are selected using a predefined hash function to determine individual weights in the dynamic parameter layer. The proposed network-joint network with the CNN for ImageQA and the parameter prediction network-is trained end-to-end through back-propagation, where its weights are initialized using a pre-trained CNN and GRU. The proposed algorithm illustrates the state-of-the-art performance on all available public ImageQA benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Trevor Darrell
  3. Jianfeng Gao
  4. Dacheng Tao
  5. Yoshua Bengio
  6. Percy Liang
  7. Sergey Levine
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Chunhua Shen

In Multimodal Learning

  1. Anton van den Hengel
  2. Christopher Kanan
  3. Kushal Kafle
  4. Qi Wu
  5. Marcus Rohrbach
  6. Trevor Darrell
  7. Devi Parikh
  8. Damien Teney

In Question Answering & Reading Comprehension

  1. Anton van den Hengel
  2. Christopher Kanan
  3. Kushal Kafle
  4. Damien Teney
  5. Qi Wu
  6. Chunhua Shen
  7. Peng Wang
  8. Anthony R. Dick

Citations in the corpus