Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes

By Caglar Gulcehre, A. P. Sarath Chandar, Kyunghyun Cho, Yoshua Bengio. 2016

66 citations recorded by the source corpus.

We extend neural Turing machine (NTM) model into a dynamic neural Turing machine (D-NTM) by introducing a trainable memory addressing scheme. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies including both linear and nonlinear ones. We implement the D-NTM with both continuous, differentiable and discrete, non-differentiable read/write mechanisms. We investigate the mechanisms and effects of learning to read and write into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRUcontroller. The D-NTM is evaluated on a set of Facebook bAbI tasks and shown to outperform NTM and LSTM baselines. We have done extensive analysis of our model and different variations of NTM on bAbI task. We also provide further experimental results on sequential pMNIST, Stanford Natural Language Inference, associative recall and copy tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “…explanations either in the form of the language (Ling et al., 2017; Hendricks et al., 2016), attention (Bahdanau et al., 2014; Mnih et al., 2014; Gulcehre et al., 2016; Malinowski et al., 2018; Xu and Saenko, 2016; Yang et al., 2016), phrase selection (Lei et al., 2016), distillation into…”
Tracking the World State with Recurrent Entity Networks (2016, background): “…the larger MemN2N reported in the appendix of (Sukhbaatar et al., 2015), the Dynamic Memory Network of (Xiong et al., 2016), the Dynamic Neural Turing Machine (Gulcehre et al., 2016), the Neural Turing Machine (Graves et al., 2014) and the Differentiable NeuralComputer (Graves et al., 2016).”
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension (2016, background): “…memory enhanced neural networks like Neural Turing Machines (Graves, Wayne, and Danihelka 2014) and its variants (Zhang, Yu, and Zhou 2015; Gulcehre et al. 2016; Zaremba and Sutskever 2015) were also potential candidates for the task, and Gulcehre et al. (2016) reported results on the…”
Inductive biases for deep learning of higher-level cognition (2020, methodology): “, 2020) and is the standard tool for memory-augmented neural networks (Graves et al., 2014; Sukhbaatar et al., 2015; Gulcehre et al., 2016; Santoro et al., 2018).”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “There have been various eﬀorts to address this shortcoming in visual attention [15], attention to text [16], and more general machine learning domains [17,18,19], but this is still a very active area of research.”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “In a high level, the proposed TM-NMT first stores each target symbol of each retrieved translation pair into a key-value memory (Gulcehre et al., 2016b; Miller et al., 2016).”
Recurrent Memory Transformer (2022, background): “MANN architectures with a more advanced addressing mechanisms such as address-content separation and multi-step addressing were proposed in (Gulcehre et al., 2016, 2017; Meng and Rumshisky, 2018).”
Deep Reinforcement Learning: An Overview (2017, background): “There are endeavours for hard attention (Gulcehre et al., 2016; Liang et al., 2017a; Luo et al., 2016; Xu et al., 2015; Zaremba and Sutskever, 2015), which is the way conventional computers access memory.”
Learning Graphical State Transitions (2016, background): “…et al., 2015), Recurrent Entity Network (EntNet, Henaff et al., 2016), Neural Turing Machine (NTM, Graves et al., 2014), Dynamic NTM (D-NTM, Gulcehre et al., 2016), a larger version of the MemN2N model with weight tying and nonlinearity (MemN2N*, Sukhbaatar et al., 2015), Differentiable…”
Memory Transformer (2020, background): “MANN architectures with a more advanced addressing mechanisms such as address-content separation and multi-step addressing were proposed in [10, 9, 18].”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, methodology): “We hypothesize that such overloaded use of output representations makes training the model difﬁcult and propose a modiﬁcation to the attention mechanism which separates these functions explicitly, inspired by Miller et al. (2016); Ba et al. (2016); Reed & de Freitas (2015); Gulcehre et al. (2016).”
Implementing Neural Turing Machines (2018, methodology): “But, MANNs have been shown to outperform LSTMs on several artificial sequence learning tasks that require a large memory and/or complicated memory access patterns, for example memorization of long sequences and graph traversal [4,5,6,11].”
Latent Alignment and Variational Attention (2018, methodology): “5 Hard Attention Hard attention is an approximate inference approach for latent alignment (Figure 2a) [80, 4, 53, 26].”