Policy Learning with Continuous Memory States for Partially Observed Robotic Control

By Marvin Zhang, Sergey Levine, Zoe McCarthy, Chelsea Finn, Pieter Abbeel. 2015

19 citations recorded by the source corpus.

Policy learning for partially observed control tasks requires policies that have the ability to store information from past observations. In this paper, we present a method for learning policies with memory for high-dimensional, continuous systems. Our approach does not assume a known state representation and does not attempt to explicitly model the belief over the unobserved state. Instead, we directly learn a policy that can read and write from an internal continuous-valued memory. This type of policy can be interpreted as a type of recurrent neural network (RNN). However, our approach avoids many of the common problems that plague RNNs, such as the vanishing and exploding gradient issues, by instead representing the memory as state variables. The policy is then optimized by using a guided policy search algorithm that alternates between optimizing trajectories through state space (including both physical and memory states), and training a policy with supervised learning to match these trajectories. We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Memory-based control with recurrent neural networks (2015, methodology): “[30] trained LSTM-based stochastic policies using Reinforce; [31, 29, 32] used actor-critic architectures.”
Deep Variational Reinforcement Learning for POMDPs (2018, methodology): “There are several approaches that utilise RNN s in POMDP s (Bakker, 2002; Wierstra et al., 2007; Zhang et al., 2015; Heess et al., 2015), including multi-agent settings (Foerster et al., 2016), learning text-based fantasy games (Narasimhan et al., 2015) or, most recently, applied to Atari…”
Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning (2017, background): “The recent marriage between RL and Neural Networks (Deep RL) opened a number of new opportunities for control in non-linear dynamical systems [19, 20, 37].”