Learning to Transduce with Unbounded Memory

By Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, Phil Blunsom. 2015

305 citations recorded by the source corpus.

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, background): “Let x j be an embedding for each source symbol; our three variants of the source representation ˆ x j are: (a) no atten , just symbol embeddings by themselves, i.e. ˆ x j = x j ; (b) simple attention, symbol embeddings and soft-pairing for each symbol, i.e. ˆ x j = [ x j ; c j ] where c j = =1…”
Adaptive Computation Time for Recurrent Neural Networks (2016, methodology): “A similar distribution was recently used to define differentiable push and pop operations for neural stacks and queues [11].”
Improving Neural Language Models with a Continuous Cache (2016, background): “In the context of sequence prediction, several memory augmented neural networks have obtained promising results (Sukhbaatar et al., 2015; Graves et al., 2014; Grefenstette et al., 2015; Joulin & Mikolov, 2015).”
Programmable Agents (2017, background): “Different approaches to this problem focus on networks that infer programs [3, 10, 22], on networks that execute programs [39, 30, 5] and on networks that jointly do both [14, 17, 38, 29, 13]; although there is much overlap between many of these approaches.”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Notable examples of such tasks are: (1) Symbolic manipulation (Grefenstette et al., 2015; Joulin & Mikolov, 2015; Kaiser & Sutskever, 2016; Kurach et al., 2016; Graves et al., 2016; Dong et al., 2019), which usually involves manipulating lists of ∗Correspondence to: Denny Zhou <dennyzhou@google.”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, methodology): “To examine how well our conclusions generalise to different machine reading tasks and language styles, we also tested the best-performing Memory Networks on the CNN QA task (Hermann et al., 2015).”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, background): “Deterministic discreteness can be relaxed and approximated reasonably well with sigmoidal functions or the softmax [15, 13], but, if a distribution over discrete states is needed, there is no clear solution.”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “This makes it difﬁcult to distinguish simple but robust sequence models from more expressive architectures (Socher, 2014; Grefenstette et al., 2015; Joulin and Mikolov, 2015).”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, background): “Possessing an inﬁnite Markovian structure (and thus an inﬁnite receptive ﬁelds) equips them to model sequential data (Elman, 1990), especially natural language (Grefenstette et al., 2015) effectively.”
Reasoning about Entailment with Neural Attention (2015, methodology): “To encourage such behavior we employ neural word-by-word attention similar to Bahdanau et al. (2015), Hermann et al. (2015) and Rush et al. (2015).”
Neural Random Access Machines (2015, methodology): “Other models with this property include variants of Stack-Augmented recurrent neural networks (Joulin & Mikolov, 2015; Grefenstette et al., 2015), and the Grid-LSTM (Kalchbrenner et al., 2015)—of which the Grid-LSTM has achieved the greatest success on both synthetic and real tasks.”
DeepCoder: Learning to Write Programs (2016, background): “t can write computer programs. Recently, there has been much interest in program-like neural network models (Graves et al., 2014; Weston et al., 2015; Kurach et al., 2015; Joulin & Mikolov, 2015; Grefenstette et al., 2015; Sukhbaatar et al., 2015; Neelakantan et al., 2016; Kaiser & Sutskever”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Other methods have attempted to capture different types of structure by mimicking key hardware and software components in computers and how they transfer information between each other, such as persistent slotted storage, registers, memory I/O controllers, stacks, and queues (e.g. Dyer et al., 2015;”
Neural Semantic Encoders (2016, background): “Researchers have also explored ways to add unbounded memory to LSTMs [7].”