Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

By A. Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, Richard Socher. 2015

1,220 citations recorded by the source corpus.

Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to-end and obtains state-of-the-art results on several types of tasks and datasets: question answering (Facebook's bAbI dataset), text classification for sentiment analysis (Stanford Sentiment Treebank) and sequence modeling for part-of-speech tagging (WSJ-PTB). The training for these different tasks relies exclusively on trained word vector representations and input-question-answer triplets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “…Models for MRC Currently, end-to-end neural network models have achieved great successes for machine reading comprehension (Seo et al. 2016; Kumar et al. 2015; Sukhbaatar et al. 2015; Cui et al. 2016; Xiong, Zhong, and Socher 2016; Dhingra et al. 2016; Shen et al. 2016; Hu et al. 2017;…”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “MVN [Guo et al., 2017] 51.5 T R E C - 50 SVM [Loni et al., 2011] 89.0 DMN [Kumar et al., 2016] 52.1 SNoW [Li and Roth, 2006] 89.3 LSTM-CNN [Zhou et al., 2016] 52.4 BCN+Char+CoVe (Ours) 90.2 TE-LSTM [Huang et al., 2017] 52.6 RulesUHC [da Silva et al., 2011] 90.8 NTI [Munkhdalai and Yu, 2016b] 53…”
Universal Transformers (2018, background): “Similar to dynamic memory networks (Kumar et al., 2016), there is an iterative attention process in UTs that allows the model to condition its attention over memory on the result of previous iterations.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “We frame all tasks as question answering [Kumar et al., 2016] by allowing task specification to take the form of a natural language question q: all inputs have a context, question, and answer (Fig.”
Generating Sentences from a Continuous Space (2015, methodology): “This technique is a variant of word dropout (Iyyer et al., 2015; Kumar et al., 2016), applied not to a feature extractor but to a decoder.”
Pointer Sentinel Mixture Models (2016, background): “…to increase capacity and minimize the impact of vanishing gradients, some language and translation mod-Pointer els have also added a soft attention or memory component (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Cheng et al., 2016; Kumar et al., 2016; Xiong et al., 2016; Ahn et al., 2016).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “The way we ask a model to respond to instructions is similar to QA-based task formulation (Kumar et al., 2016; McCann et al., 2018), which aims to unify NLP tasks by casting them as QA over a context.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “This is achieved by unifying all the tasks into a common format (Kumar et al., 2016; Khashabi et al., 2020) via instructions, and training them together by sharing all the weights of the model across all tasks.”
Coarse-to-Fine Question Answering for Long Documents (2016, positive): “Current state-of-the-art approaches for QA over documents are based on recurrent neural networks (RNNs) that encode the document and the ques-Query tion to determine the answer (Hermann et al., 2015; Chen et al., 2016; Kumar et al., 2016; Kadlec et al., 2016; Xiong et al., 2016).”
Gated Graph Sequence Neural Networks (2015, background): “GGS-NNs are related to soft alignment and attentional models (e.g., Bahdanau et al. (2014); Kumar et al. (2015); Sukhbaatar et al. (2015)) in two respects: ﬁrst, the graph representation in Eq.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Deep convolutional neural networks (DCNNs) have achieved great success on many tasks across a variety of domains, such as vision [41, 64, 26, 25, 8], audition [29, 23, 12, 51], and natural language processing [73, 32, 10, 42].”
Explainable Multi-hop Verbal Reasoning Through Internal Monologue (2021, background): “, 2015), Dynamic Memory Networks (DMN) (Kumar et al., 2016) and Compositional Attention”
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2021, background): “There has for some time been attempts to design a uniﬁed interface for NLP tasks (Kumar et al., 2016; Raffel et al., 2020).”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “Knowledge-based question answering has received much more attention in the NLP community (e.g., [5, 50, 49, 6, 40, 28, 7]).”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, methodology): “Sukhbaatar et al. (2015) and Kumar et al. (2015) trained Memory Networks with RNN components end-to-end with soft memory access, and applied them to additional language tasks.”
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task (2016, background): “fter reading a sequence of sentences “John is in the playground; Bob is in the ofﬁce; John picked up the football; Bob went to the kitchen.” Various types of memory networks (Sukhbaatar et al., 2015; Kumar et al., 2016) have been shown effective on these tasks, and Lee et al. (2016) show that vector”