Memory-based control with recurrent neural networks

By Nicolas Manfred Otto Heess, Jonathan J. Hunt, Timothy P. Lillicrap, David Silver. 2015

333 citations recorded by the source corpus.

Partially observed control problems are a challenging aspect of reinforcement learning. We extend two related, model-free algorithms for continuous control -- deterministic policy gradient and stochastic value gradient -- to solve partially observed domains using recurrent neural networks trained with backpropagation through time. We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements. These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps. We also demonstrate success on a combined exploration and memory problem in the form of a simplified version of the well-known Morris water maze task. Finally, we show that our approach can deal with high-dimensional observations by learning directly from pixels. We find that recurrent deterministic and stochastic policies are able to learn similarly good solutions to these tasks, including the water maze where the agent must learn effective search strategies.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, methodology): “The techniques we described here for scaling DPG are also applicable to stochastic policies by using the reparametrization trick (Heess et al., 2015; Schulman et al., 2015a).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…works have studied applying sequence models of various types to represent components in standard RL algorithms, such as policies, value functions, and models (Bakker, 2002; Heess et al., 2015a; Chiappa et al., 2017; Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021; Kumar et al., 2020b).”
Guided Deep Reinforcement Learning for Swarm Systems (2017, background): “There exist extensions to deal with partial observability [1], [3], but, so far, they are only formulated for single-agent scenarios.”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, methodology): “Its extension to recurrent policies, Recurrent Deterministic Policy Gradient (RDPG) [36], provides a method to train recurrent policies with off-policy data.”
Cognitive Mapping and Planning for Visual Navigation (2017, neutral): “Follow-up works (Mnih et al. 2016; Gu et al. 2016; Schulman et al. 2015) propose improvements to DRL algorithms, (Oh et al. 2016; Mnih et al. 2016; Wierstra et al. 2010; Heess et al. 2015; Zhang et al. 2016) study how to incorporate memory into such neural network based models.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “While prior work has studied methods that can train recurrent Q-functions with off-policy Q-learning methods, such methods have often been applied to much simpler tasks (Heess et al., 2015), and in discrete environments (Hausknecht & Stone, 2015).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “deep learning is to utilise recurrent neural networks (RNNs) [163, 44, 45, 85, 96], which, unlike feedforward neural networks, are dynamical systems.”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, methodology): “Indeed, previous work [12] has shown that a recurrent network can learn to control a dynamic system with unknown”
Emergent Coordination Through Competition (2019, methodology): “A.1 POLICY UPDATES The Stochastic Value Gradients (SVG0) algorithm used throughout this work is a special case of the family of policy gradient algorithms provided by Heess et al. (2015b) in which the gradient of a value function used to compute the policy gradient, and is closely related to the Det”
Deep Variational Reinforcement Learning for POMDPs (2018, methodology): “In this section, we formalise POMDPs and provide background on recent advances in VAEs that we use.”
Robust Adversarial Reinforcement Learning (2017, background): “Recent applications of deep reinforcement learning (deep RL) have shown great success in a variety of tasks ranging from games (Mnih et al., 2015; Silver et al., 2016), robot control (Gu et al., 2016; Lillicrap et al., 2015; Mor-datch et al., 2015), to meta learning (Zoph & Le, 2016).”
Autonomous vehicle control via deep reinforcement learning (2017, background): “This section will introcude the reader to the basic theory in RL [27] and recent algorithms that are relevant for the thesis [5][28][6].”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “Indeed, a number of prior works have studied applying sequence models of various types to represent components in standard RL algorithms, such as policies, value functions, and models [4, 21, 9, 45, 44, 34].”
Sim2Real View Invariant Visual Servoing by Recurrent Control (2017, background): “The use of recurrent networks for control has previously been explored in a number of works on reinforcement learning, including methods for navigation [22, 24], continuous control [13, 38], and simulation to real world transfer [25].”
Learning Multi-Agent Communication through Structured Attentive Reasoning (2020, background): “(2) Extend R-MADDPG [26] from a single step transition update to a recurrent trajectory-based training [27] through TD3.”
Imitating driver behavior with generative adversarial networks (2017, background): “policies [30], [31] disambiguate perceptually similar states by acting with respect to histories of, rather than individual, observations.”
Autonomous Navigation of UAVs in Large-Scale Complex Environments: A Deep Reinforcement Learning Approach (2019, methodology): “The proposed algorithm is demonstrated in Table I, where the use of two target actor and critic networks and a replay memory aim to stabilize the learning process, and the addition of the stochastic process (exploration noise) to the currently learned policy is to encourage agents keep exploring the”