Memory Networks
By Jason Aaron Edward Weston, Sumit Chopra, Antoine Bordes. 2014
1,867 citations recorded by the source corpus.
Abstract: We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response. We evaluate them on a large-scale QA task, and a smaller, but more complex, toy task generated from a simulated world. In the latter, we show the reasoning power of such models by chaining multiple supporting sentences to answer questions that require understanding the intension of verbs.
Subfields: Electrical and Electronic Engineering, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Electrical and Electronic Engineering
- Yoshua Bengio
- J. Urgen Schmidhuber
- Sergey Levine
- Yuxi Li
- Oriol Vinyals
- Joelle Pineau
- Razvan Pascanu
- Timothy P. Lillicrap
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Danqi Chen
- Jason Aaron Edward Weston
- Wen-tau Yih
- William W. Cohen
- Minjoon Seo
- Jianfeng Gao
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Jason Aaron Edward Weston
- Kyunghyun Cho
- Percy Liang
- Yejin Choi
- Graham Neubig
Citations in the corpus
- Structured Attention Networks (2017, methodology): “This approach alleviates the bottleneck of compressing a source into a fixed-dimensional vector by equipping a model with variable-length memory (Weston et al., 2014; Graves et al., 2014; 2016), thereby providing random access into the source as needed.”
- Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Memory Networks [45] have a similarity to the transformer model in design as it also has layers of attention for processing a fix memory representing the input document and has been shown to be effective for language modeling in [39].”
- Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Unlike previously popular attention mechanisms (Weston et al., 2015; Hill et al., 2016; Sordoni et al., 2016; Shen et al., 2016), the attention flow layer is not used to summarize the query and context into single feature vectors.”
- A large annotated corpus for learning natural language inference (2015, background): “…effective at modeling simple dimensions of meaning like evaluative sentiment (e.g., Socher et al. 2013), but it is less clear that they can be trained to support the full range of logical and commonsense inferences required for NLI (Bowman et al., 2015; Weston et al., 2015b; Weston et al., 2015a).”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Memory-based Architectures Our document index can be seen as a large external memory for neural networks to attend to, analagous to memory networks [58, 48].”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “…our work share motivations with works such as product key memory (Lample et al., 2019), which enables sub-linear memory access in a memory network (Weston et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015), allowing these scalable memory layers to be integrated into large language models.”
- Graph Attention Networks (2017, background): “Other related approaches include locally linear embedding (LLE) (Roweis & Saul, 2000) and memory networks (Weston et al., 2014).”
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m nτ ∈ R M × d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014; Weston et al., 2014).”
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, background): “It has stimulated work on use of memory network neural architectures (Weston et al., 2014), supporting a form of multistep reasoning where a neural memory propagates information from one step to another (e.g. Henaff et al., 2016; Seo et al., 2017a).”
- Matching Networks for One Shot Learning (2016, methodology): “A key component which allowed for more expressive models was the introduction of “content” based attention in [2], and “computer-like” architectures such as the Neural Turing Machine [4] or Memory Networks [29].”
- Teaching Machines to Read and Comprehend (2015, background): “While obtaining supervised natural language reading comprehension data has proved difficult, some researchers have explored generating synthetic narratives and queries [3, 4].”
- Reformer: The Efficient Transformer (2020, methodology): “The original implementation of memory networks (Weston et al., 2014) and later work on scaling it (Bordes et al., 2015; Chandar et al., 2016) used memory with size in the millions.”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “This line of investigation proved to be the first successful demonstration of multiple layers of attention for reasoning with text (Weston, Chopra, and Bordes, 2014; Sukhbaatar et al., 2015) which has now become part of the defacto method (Vaswani et al., 2017).”
- Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely tied to memory-based neural layers (Weston et al., 2014; Sukhbaatar et al., 2015).”
- Large Memory Layers with Product Keys (2019, background): “For instance, memory-based neural layers [45, 41] are an efficient way to represent variable length inputs for complex problems such as question answering [46].”
- Reading Wikipedia to Answer Open-Domain Questions (2017, background): “…progress recently thanks to new deep learning architectures like attention-based and memory-1 augmented neural networks (Bahdanau et al., 2015; Weston et al., 2015; Graves et al., 2014) and release of new training and evaluation datasets like QuizBowl (Iyyer et al., 2014), CNN/Daily Mail…”
- A Knowledge-Grounded Neural Conversation Model (2017, methodology): “Memory network models are widely used in Question Answering to make inferences based on the facts saved in the memory (Weston et al., 2015).”
- Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “For LSTM, we use the numbers published in Weston et al. (2014).”
- Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index (2019, background): “Memory networks One can view the phrase index as an external memory (Weston et al., 2015; Miller et al., 2016) where the key is the phrase vector and the value is the corresponding answer phrase span.”