Key-Value Memory Networks for Directly Reading Documents

By Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, Jason Aaron Edward Weston. 2016

1,081 citations recorded by the source corpus.

Directly reading documents and being able to answer questions from them is an unsolved challenge. To avoid its inherent difficulty, question answering (QA) has been directed towards using Knowledge Bases (KBs) instead, which has proven effective. Unfortunately KBs often suffer from being too restrictive, as the schema cannot support certain types of answers, and too sparse, e.g. Wikipedia contains much more information than Freebase. In this work we introduce a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation. To compare using KBs, information extraction or Wikipedia documents directly in a single framework we construct an analysis tool, WikiMovies, a QA dataset that contains raw text alongside a preprocessed KB, in the domain of movies. Our method reduces the gap between all three settings. It also achieves state-of-the-art results on the existing WikiQA benchmark.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “We found adding e j to be beneﬁcial and it resembles key-value memory networks where the keys are the z uj and the values are the z uj + e j (Miller et al., 2016).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “Numerous prior works have demonstrated the benefit of adding a discrete retrieval step to neural networks (Miller et al., 2016; Chen et al., 2017), but did not apply the framework to language model pre-training and employed nonlearned retrievers to handle large-scale document collections.”
Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “Universal Lexical Representation (ULR) We tackle this issue by dynamically building a vocabulary speciﬁc to each language using a key-value memory network (Miller et al., 2016; Gul-cehre et al., 2018), as was done successfully for low-resource machine translation recently by Gu et al. (2018b).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, neutral): “Secondly, we also compare to recently published models from See et al. (2019). The authors studied the effects of controllable generation.”
Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, methodology): “For the retriever, we use the Key-Value Memory Network (Miller et al., 2016) already shown to perform well for this dataset (Zhang et al., 2018), which attends over the dialogue history, to learn input and candidate retrieval embeddings that match using cosine 1 http://convai.io/ similarity.”
Reading Wikipedia to Answer Open-Domain Questions (2017, background): “WikiMovies This dataset, introduced in (Miller et al., 2016), contains 96k question-answer pairs in the domain of movies.”
Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “In fact, such models usually truncate the documents and consider only a limited number of tokens (Miller et al., 2016; Hewlett et al., 2016).”
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index (2019, background): “Memory networks One can view the phrase index as a fixed external memory (Miller et al., 2016) where the key is the phrase vector and the value is the corresponding answer phrase span.”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “Iterative Reasoning has shown signiﬁcant improvements in models using memory networks (Sukhbaatar et al., 2015; Miller et al., 2016) for question answering in text and knowledge bases (Das et al., 2017; Yang et al., 2017; Das et al., 2018).”
Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering (2018, methodology): “WikiMovies (Miller et al., 2016) contains questions regarding movies collected from OMDb and the MovieLens database.”
Learning Through Dialogue Interactions (2016, methodology): “, 2015; Weston, 2016), using either a given database of knowledge (Bordes et al., 2015; Miller et al., 2016) or short stories (Weston et al.”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “To understand the unstructured documents, the Key-Value Memory Network (KV-MemNN) [13] performs reasoning by utilizing different encodings for key memory and value memory.”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, background): “The key-value (KV) memory network (Miller et al., 2016) was proposed as an improvement to the memory network by performing attention over keys and outputting the values (instead of the same keys as in the original), which can outperform memory networks dependent on the task and definition of the key”
Compositional Attention Networks for Machine Reasoning (2018, methodology): “Note that while we compute the attention based on the control states, we use it to average over the memory states, in a way that resembles Key-Value Memory Networks (Miller et al., 2016).”
Dialogue Learning With Human-In-The-Loop (2016, neutral): “…the line of research that focuses on supervised learning for question answering (QA) from dialogues (Dodge et al., 2015; Weston, 2016), either given a database of knowledge (Bordes et al., 2015; Miller et al., 2016) or short texts (Weston et al., 2015; Hermann et al., 2015; Rajpurkar et al., 2016).”
Quasar: Datasets for Question Answering by Search and Reading (2017, methodology): “However, even the largest KBs are often incomplete (Miller et al., 2016; West et al., 2014), and hence can only answer a limited subset of all possible factoid questions.”
RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems (2018, background): “Memory networks [17, 25, 38] is a recurrent attention model that utilizes an external memory module for question answering and language modeling.”