Training Recurrent Answering Units with Joint Loss Minimization for VQA

By Hyeonwoo Noh, Bohyung Han. 2016

73 citations recorded by the source corpus.

We propose a novel algorithm for visual question answering based on a recurrent deep neural network, where every module in the network corresponds to a complete answering unit with attention mechanism by itself. The network is optimized by minimizing loss aggregated from all the units, which share model parameters while receiving different information to compute attention probability. For training, our model attends to a region within image feature map, updates its memory based on the question and attended image feature, and answers the question based on its memory state. This procedure is performed to compute loss in each step. The motivation of this approach is our observation that multi-step inferences are often required to answer questions while each problem may have a unique desirable number of steps, which is difficult to identify in practice. Hence, we always make the first unit in the network solve problems, but allow it to learn the knowledge from the rest of units by backpropagation unless it degrades the model. To implement this idea, we early-stop training each unit as soon as it starts to overfit. Note that, since more complex models tend to overfit on easier questions quickly, the last answering unit in the unfolded recurrent neural network is typically killed first while the first one remains last. We make a single-step prediction for a new question using the shared model. This strategy works better than the other options within our framework since the selected model is trained effectively from all units without overfitting. The proposed algorithm outperforms other multi-step attention based approaches using a single step prediction in VQA dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “In [55], an increase of 2% was observed by using ResNet-101 instead of the VGG-16 CNN for image features.”
An Analysis of Visual Question Answering Algorithms (2017, methodology): “They were shown random images from COCO and Visual Genome and could also upload images.”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, background): “As these questions concern the possible usages of objects present in the scene (such as Can you eat RAU* MCB* QTA MuRel [35] [16] [40]”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “There are different approaches have been proposed for VQA [18, 5, 8, 45, 20, 42, 24, 2, 28, 23, 38, 46, 29, 40].”
Learning to Specialize with Knowledge Distillation for Visual Question Answering (2018, methodology): “incorporate various techniques to concentrate only on the tasks specified by questions, including attention mechanism [26, 27, 31, 33, 34], multimodal fusion schemes [4, 8, 16], adaptive networks [28] or modular networks [2, 3, 13].”
PlotQA: Reasoning over Scientific Plots (2019, background): “Models: The availability of the above mentioned datasets has facilitated the development of complex end-to-end neural network based models ([36], [22], [37], [25], [30], [12], [33]).”
DVQA: Understanding Data Visualizations via Question Answering (2018, methodology): “Numerous VQA algorithms have been proposed, ranging from Bayesian approaches [12, 24], methods using spatial attention [32, 31, 23, 25], compositional approaches [2, 3], and bilinear pooling schemes [20, 6].”
Progressive Attention Networks for Visual Attribute Prediction (2016, background): “Many visual attention models have been introduced in the previous literature, and they have shown that attaching an attention to NNs improves the accuracy in various tasks such as image classification [2, 11, 13], image generation [10], image caption generation [23] and visual question answering [17”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”
Visual question answering: A survey of methods and datasets (2016, unknown): “Joint-Loss [57] X X classification ResNet [25]”