End-To-End Memory Networks

By Sainbayar Sukhbaatar, Arthur Szlam, Jason Aaron Edward Weston, Rob Fergus. 2015

2,743 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, background): “End-to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [28].”
Convolutional Sequence to Sequence Learning (2017, background): “This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al.”
Structured Attention Networks (2017, methodology): “…2015), image captioning (Xu et al., 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al., 2015; Sukhbaatar et al., 2015), and algorithm-learning (Graves et al., 2014; Vinyals et al., 2015), among many other applications (see Cho et al. (2015)…”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “Recent work attempts to address the latter limitation using external memories (Weston et al., 2015; Sukhbaatar et al., 2015; Grefenstette et al., 2015).”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “…MRC Currently, end-to-end neural network models have achieved great successes for machine reading comprehension (Seo et al. 2016; Kumar et al. 2015; Sukhbaatar et al. 2015; Cui et al. 2016; Xiong, Zhong, and Socher 2016; Dhingra et al. 2016; Shen et al. 2016; Hu et al. 2017; Wang, Yan, and Wu…”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Memory Networks [45] have a similarity to the transformer model in design as it also has layers of attention for processing a ﬁx memory representing the input document and has been shown to be effective for language modeling in [39].”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Memory-based Architectures Our document index can be seen as a large external memory for neural networks to attend to, analagous to memory networks [58, 48].”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “…our work share motivations with works such as product key memory (Lample et al., 2019), which enables sub-linear memory access in a memory network (Weston et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015), allowing these scalable memory layers to be integrated into large language models.”
Generating Long Sequences with Sparse Transformers (2019, background): “Other works have investigated attention with multiple ”hops”, such as (Sukhbaatar et al., 2015) and (Gehring et al., 2017).”
Universal Transformers (2018, background): “Another related model architecture is that of end-to-end Memory Networks (Sukhbaatar et al., 2015).”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, methodology): “5 Our own hunch is that, despite the initially disappointing results of the “vanilla” Memory Network we tested, the ability to store information in a longer-term memory will be a crucial component of successful models, coupled with the ability to perform some kind of reasoning about what’s stored in”
Teaching Machines to Read and Comprehend (2015, background): “n to exploit document structure. While obtaining supervised natural language reading comprehension data has proved difﬁcult, some researchers have explored generating synthetic narratives and queries [3, 4]. Such approaches allow the generation of almost unlimited amounts of supervised data and enab”
Pointer Sentinel Mixture Models (2016, background): “…to increase capacity and minimize the impact of vanishing gradients, some language and translation mod-Pointer els have also added a soft attention or memory component (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Cheng et al., 2016; Kumar et al., 2016; Xiong et al., 2016; Ahn et al., 2016).”
Evaluating Large Language Models Trained on Code (2021, methodology): “…incorporated inductive biases based on modern computing devices, such as the Neural Turing Machine (Graves et al., 2014), memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), the Neural GPU (Kaiser & Sutskever, 2015), and the differentiable neural computer (Graves et al., 2016).”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Our model builds on the work of Gehring et al. (2017b) who develop an encoder-decoder architecture for machine translation with an attention mechanism ( Sukhbaatar et al., 2015) based exclusively on deep convolutional networks.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “This line of investigation proved to be the first successful demonstration of multiple layers of attention for reasoning with text (Weston, Chopra, and Bordes, 2014; Sukhbaatar et al., 2015) which has now become part of the defacto method (Vaswani et al., 2017).”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely tied to memory-based neural layers (Weston et al., 2014; Sukhbaatar et al., 2015).”
Large Memory Layers with Product Keys (2019, background): “For instance, memory-based neural layers [45, 41] are an efﬁcient way to represent variable length inputs for complex problems such as question answering [46].”