Neural Turing Machines

By Alex Graves, Greg Wayne, Ivo Danihelka. 2014

2,582 citations recorded by the source corpus.

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-toend, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Structured Attention Networks (2017, methodology): “…et al., 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al., 2015; Sukhbaatar et al., 2015), and algorithm-learning (Graves et al., 2014; Vinyals et al., 2015), among many other applications (see Cho et al. (2015) for a comprehensive review).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “…our work share motivations with works such as product key memory (Lample et al., 2019), which enables sub-linear memory access in a memory network (Weston et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015), allowing these scalable memory layers to be integrated into large language models.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Thus, we can cache a predeﬁned length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m nτ ∈ R M × d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014; Weston et al., 2014).”
Universal Transformers (2018, methodology): “…practical sequence models competitive on large-scale tasks such as machine translation, and computationally universal models such as the Neural Turing Machine or the Neural GPU (Graves et al., 2014; Kaiser & Sutskever, 2016), which can be trained using gradient descent to perform algorithmic tasks.”
Learning to Optimize Neural Nets (2017, methodology): “…syntax trees using genetic algorithms, Liang et al. (2010) deﬁnes a hierarchical Bayesian prior over programs in a formal language and infers the best program using approximate sampling and neural Turing machines (Graves et al., 2014) try to predict output examples from input examples directly.”
Learning to learn by gradient descent by gradient descent (2016, methodology): “Conveniently, this structure of interaction with a large dynamically updated state corresponds in a fairly direct way to the architecture of a Neural Turing Machine (NTM), where M t corresponds to the NTM memory [Graves et al., 2014].”
Matching Networks for One Shot Learning (2016, background): “A key component which allowed for more expressive models was the introduction of “content” based attention in [2], and “computer-like” architectures such as the Neural Turing Machine [4] or Memory Networks [29].”
Adaptive Computation Time for Recurrent Neural Networks (2016, methodology): “ACT promises to be particularly interesting for recurrent architectures containing soft attention modules [2, 10, 34, 12], which it could enable to dynamically adapt the number of glances or internal operations they perform at each time-step.”
Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, neutral): “The related problem of neural algorithm induction has attracted considerable interest (Graves et al., 2014; Kurach et al., 2016; Kaiser & Sutskever, 2016; Graves et al., 2016; Reed & de Freitas, 2016; Veliˇckovi´c et al., 2020a;b).”
Program Synthesis with Large Language Models (2021, neutral): “…are deep network architectures that aim to learn algorithms from input-output examples, by structuring the network in a way that corresponds to mathematical models of computation like Turing machines [Graves et al., 2014, Kaiser and Sutskever, 2016, Kurach et al., 2016, Graves et al., 2016].”
Evaluating Large Language Models Trained on Code (2021, methodology): “Later attempts at program induction incorporated inductive biases based on modern computing devices, such as the Neural Turing Machine (Graves et al., 2014), memory networks (Weston et al., 2015; Sukhbaatar et al., 2015), the Neural GPU (Kaiser & Sutskever, 2015), and the differentiable neural compu”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “Memory augmented neural networks with sparse reads and writes have also been proposed in (Rae et al., 2016) as a way to scale Neural Turing Machines (Graves et al., 2014).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “While in machine learning some read, write memory architectures have been developed (Graves, Wayne, and Danihelka, 2014; Henaff et al., 2017) they have mostly not been successful so far at scaling to realistic large-scale dialogue tasks.”
Large Memory Layers with Product Keys (2019, background): “Such memories can also operate in feature space and have various reading and writing mechanisms [22, 16].”
Reading Wikipedia to Answer Open-Domain Questions (2017, background): “That subﬁeld has made considerable progress recently thanks to new deep learning architectures like attention-based and memory-1 augmented neural networks (Bahdanau et al., 2015; Weston et al., 2015; Graves et al., 2014) and release of new training and evaluation datasets like QuizBowl (Iyyer et al.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “At the same time, our model seems simpler than many other models used for this type of tasks (Tabor, 2000; Bod´en & Wiles, 2000; Graves et al., 2014).”