Multimodal Residual Learning for Visual QA

By Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang. 2016

319 citations recorded by the source corpus.

Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Residual Attention Network for Image Classification (2017, background): “Similar to our work, [25, 29, 21, 18] use residual learning with attention mechanism to benefit from residual learning.”
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, background): “The latter one aims to ﬁnd region-level details by spatial attention [2,12,26,29].”
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering (2017, background): “has received a lot of attention – a number of VQA datasets have been curated [5, 21, 41, 12, 25, 11, 29, 13, 39] and a variety of deep-learning models have been developed [5, 9, 38, 37, 16, 3, 34, 19, 24, 4, 31, 20, 10, 26, 14, 35, 36, 40, 30].”
Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering (2017, background): “egories: 1) the coarse joint embedding models [6], [9], [14]; 2) the fine-grained joint embedding models with attention [8], [10], [15]–[17]; and 3) the external knowledge-based models [18], [19].”
Visual question answering: Datasets, algorithms, and future challenges (2016, methodology): “ this yielded results signicantly higher than the complex attention-based models used in [49] and [52]. Similar results have been obtained by other systems that do not employ spatial attention, e.g, [36, 64, 53]. Attention alone does not ensure good VQA performance, but incorporating attention into ”
An Analysis of Visual Question Answering Algorithms (2017, methodology): “We only use Visual Genome to make color and positional reasoning questions.”
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “Some approaches introduce more complex deep models, e.g., LSTM networks [3] or residual networks [11], to tackle the VQA task in an end-to-end fashion.”
Motion-Appearance Co-memory Networks for Video Question Answering (2018, neutral): “Many methods have been proposed [37, 5, 14, 21, 36, 7, 3, 1, 33, 23, 4, 28, 17, 13, 40, 38, 41, 42].”
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (2019, background): “In other words, DAN utilizes the excellence of deep residual learning as in (He et al., 2016; Rocktäschel et al., 2015; Yang et al., 2016; Kim et al., 2016; Vaswani et al., 2017).”
Open-Ended Video Question Answering via Multi-Modal Conditional Adversarial Networks (2020, background): “[5] propose a multimodal residual network for image question answering.”
Visual question answering: A survey of methods and datasets (2016, methodology): “The latest improvements, exemplified by MCB and MRN, still showed potential room for improvement on both the extraction of features and their projection to the embedding space.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, methodology): “Most of the methods developed for visual question answering [2,8,10,18,19,20,21,22,23,24,12,27,29,30,31,6,7,32,33,34] focus exclusively on answering questions related to observed content.”
Modulating early visual processing by language (2017, methodology): “For instance, it was shown that we can improve upon classic concatenation by an element-wise product [1, 13, 14], Tucker decomposition [2], bilinear pooling [9] or more exotic approaches [18].”
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation (2017, unknown): “Dev Standard Two-layer LSTM [3] 62.7 63.1 Region selection [38] 62.4 62.4 DPPNet [32] 62.5 62.7 MCB [9] 65.4 − Co-Attention [26] 65.9 66.1 MRN [19] 66.3 66.3 MLB [20] − 68.9 MLP + ResNet [17] 67.”