Reinforcement Learning Neural Turing Machines

By Wojciech Zaremba, Ilya Sutskever. 2015

168 citations recorded by the source corpus.

The expressive power of a machine learning model is closely related to the number of sequential computational steps it can learn. For example, Deep Neural Networks have been more successful than shallow networks because they can perform a greater number of sequential computational steps (each highly parallel). The Neural Turing Machine (NTM) [8] is a model that can compactly express an even greater number of sequential computational steps, so it is even more powerful than a DNN. Its memory addressing operations are designed to be differentiable; thus the NTM can be trained with backpropagation. While differentiable memory is relatively easy to implement and train, it necessitates accessing the entire memory content at each computational step. This makes it difficult to implement a fast NTM. In this work, we use the Re inforce algorithm to learn where to access the memory, while using backpropagation to learn what to write to the memory. We call this model the RL-NTM. Reinforce allows our model to access a constant number of memory cells at each computational step, so its implementation can be faster. The RL-NTM is the first mo del that can, in principle, learn programs of unbounded running time. We successfully trained the RL-NTM to solve a number of algorithmic tasks that are simpler than the ones solvable by the fully differentiable NTM. As the RL-NTM is a fairly intricate model, we needed a method for verifying the correctness of our implementation. To do so, we developed a simple technique for numerically checking arbitrary implementations of models that use Reinforce, which may be of independent interest.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Neural GPUs Learn Algorithms (2015, methodology): “Neural Turing Machines (Graves et al., 2014) were shown to learn a number of basic sequence transformations and memory access patterns, and their reinforcement learning variant (Zaremba & Sutskever, 2015b) has reasonable performance on a number of tasks as well.”
Learning Through Dialogue Interactions (2016, methodology): “We refer the readers to (Ranzato et al., 2015; Zaremba & Sutskever, 2015) for more details.”
Order Matters: Sequence to sequence for sets (2015, neutral): “The key element that these models utilize is a reading (or attention) mechanism to read these external memories in a fully differentiable way (though there has also been work with discrete reading mechanism, most notably RL-NTM (Zaremba & Sutskever, 2015)).”
Gradient Estimation Using Stochastic Computation Graphs (2015, background): “Combining ideas from from those two perennial topics, recent models of attention [15] and memory [29] have used networks that involve a combination of stochastic and deterministic operations.”
Self-Critical Sequence Training for Image Captioning (2016, methodology): “Using REINFORCE with a baseline b the estimate of the gradient of ∂L(θ) ∂st is given by [17]:”
Deep Reinforcement Learning for Dialogue Generation (2016, methodology): “We evaluate dialogue generation systems using both human judgments and two automatic metrics: conversation length (number of turns in the entire session) and diversity.”
Neural Programmer-Interpreters (2015, neutral): “(Zaremba & Sutskever, 2015; Zaremba et al., 2015; Kaiser & Sutskever, 2015; Kurach et al., 2015; Neelakantan et al., 2015).”
Learning to Transduce with Unbounded Memory (2015, background): “The NTM and Memory Networks [4, 12, 13] provide powerful random access memory operations, whereas we focus on a more efficient and restricted class of models which we believe are sufficient for natural language transduction tasks.”
Inferring and Executing Programs for Visual Reasoning (2017, background): “Such models can take the form of a feedforward scoring function over operators in a domain-specific language that can be used to guide program search [4], or of a recurrent network that decodes a vectorial program representation into the actual program [22, 26, 34, 46, 47, 48].”
Dialogue Learning With Human-In-The-Loop (2016, neutral): “We refer the readers to (Ranzato et al., 2015; Zaremba & Sutskever, 2015) for more details.”
Multi-Reward Reinforced Summarization with Saliency and Entailment (2018, methodology): “We use the RE-INFORCE algorithm (Williams, 1992; Zaremba and Sutskever, 2015) to learn a policy p θ de-ﬁned by the model parameters θ to predict the next action (word) and update its internal (LSTM) states.”
Neural Random Access Machines (2015, methodology): “A related model is the Reinforcement Learning Neural Turing Machine (Zaremba & Sutskever, 2015), which attempted to use reinforcement learning techniques to train a discrete-continuous hybrid model.”
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (2016, background): “A common solution to this problem is to use curriculum learning (Zaremba and Sutskever, 2015; Reed and de Freitas, 2016).”
MuProp: Unbiased Backpropagation for Stochastic Neural Networks (2015, background): “…& Tao, 2001; Peters & Schaal, 2006), probabilistic latent variable models for structured prediction, unsupervised learning of generative models (Tang & Salakhutdinov, 2013; Kingma & Welling, 2014), and most recently, attention and memory networks (Mnih et al., 2014; Zaremba & Sutskever, 2015).”
Learning to Optimize (2016, background): “Subsequent work proposes variants of this model that use different primitive memory access operations [14], more expressive operations [16, 28] or other non-differentiable operations [30, 29].”
Learning to Remember Rare Events (2017, background): “Some attempts have been made at making hard access queries to memory (Zaremba & Sutskever, 2015; Xu et al., 2015), but it was usually challenging to match the soft version.”
Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015, background): “, marginalizing over all latent stack operations), but sampling techniques and techniques from reinforcement learning have promise here (Zaremba and Sutskever, 2015), making this an intriguing avenue for future work.”
Variational Inference for Monte Carlo Objectives (2016, methodology): “…the potential pitfalls described above, estimators involving sampling from the prior have been used successfully for training models for structured output prediction (Tang & Salakhutdinov, 2013; Dauphin & Grangier, 2016) and models with hard attention (Mnih et al., 2014; Zaremba & Sutskever, 2015).”
Control of Memory, Active Perception, and Action in Minecraft (2016, background): “Zaremba & Sutskever (2015) proposed RL-NTM that has a non-differentiable memory to scale up the addressing mechanism of NTM and applied policy gradient to train the architecture.”
A Simple, Fast Diverse Decoding Algorithm for Neural Generation (2016, background): “We refer the readers to (Ranzato et al., 2015; Zaremba and Sutskever, 2015) for more details.”