Dynamic Memory Networks for Visual and Textual Question Answering

By Caiming Xiong, Stephen Merity, Richard Socher. 2016

770 citations recorded by the source corpus.

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, background): “(2016) employ a structured neural network with episodic memory modules for natural language and also visual question answering (Xiong et al., 2016).”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “We did not submit the SQuAD model for testing, but the addition of CoVe was enough to push the validation performance of the original DCN, which already used character n-gram embeddings, above the validation performance of the published version of the R-NET.”
Bidirectional Attention Flow for Machine Comprehension (2016, background): “At the coarse level of granularity, the question attends to different patches in the image (Zhu et al., 2016; Xiong et al., 2016a).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017), and the success of this approach extends to visual and textual question answering (Xiong et al., 2016).”
Pointer Sentinel Mixture Models (2016, background): “…to increase capacity and minimize the impact of vanishing gradients, some language and translation mod-Pointer els have also added a soft attention or memory component (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Cheng et al., 2016; Kumar et al., 2016; Xiong et al., 2016; Ahn et al., 2016).”
A simple neural network module for relational reasoning (2017, background): “munity [3, 22, 27, 48]. Recently, in addition to traditional symbolic-based question answering architectures, we observe a growing interest in neural-based approaches to text based question answering [34, 42, 43]. While these architectures rely on ‘memories’, we empirically show that the RN module h”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Various approaches to combine spatial image representations and sequential question representations have been proposed [7, 10, 25, 57, 86, 88, 91].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Attention-based models [12, 31, 46, 47, 48, 54] better guide the model in answering the questions by highlighting image regions that are relevant to the question.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, methodology): “uilding trust for machines among their users. 2. Related Work Visual Question Answering. A number of recent works have proposed visual question answering datasets [3,21,25, 30,10,45,37,35] and models [9,24,2,42,23,26,46, 44,43,40,34,19,28,14,41,32,16]. Our work builds on top of the VQA dataset from ”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “Dynamic Memory Networks (DMN) (Xiong et al., 2016) pool image and question with element-wise multiplication and addition, attending to part of image and question with a Episodic Memory Module (Kumar et al.”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, methodology): “We can see that image-attention-alone does improve performance over the holistic image feature (deeper LSTM Q + norm I), which is consistent with findings of previous attention models for VQA [21, 23].”
Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, background): “Recent years have seen a proliferation of datasets [19, 4] and approaches, including models based on differentiable memory [25, 24], dynamic prediction of question-specific computations [20, 2], and core improvements to the implementation of the multi-modal representation and attention mechanism [9,”
Inferring and Executing Programs for Visual Reasoning (2017, background): “Many systems for VQA employ a very similar architecture [3, 8, 9, 30, 32, 33, 44]: they combine an RNN-based embedding of the question with a convolutional network-based embedding of an image in a classification model over possible answers.”
Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions? (2016, background): “…(Bahdanau et al., 2014; Firat et al., 2016), object recognition (Ba et al., 2015; Mnih et al., 2014; Sermanet et al., 2014), image captioning (Xu et al., 2015; Cho et al., 2015) and visual question answer-∗ ing (Yang et al., 2015; Lu et al., 2016; Xu and Saenko, 2015; Xiong et al., 2016).”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “This approach has been successfully applied to various tasks including machine translation [4], text generation [16], sentence summarization [25], and question answering [15, 32].”