Deep predictive policy training using reinforcement learning

By Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, Mårten Björkman. 2017

138 citations recorded by the source corpus.

Skilled robot task learning is best implemented by predictive action policies due to the inherent latency of sensorimotor processes. However, training such predictive policies is challenging as it involves finding a trajectory of motor activations for the full duration of the action. We propose a data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations. The architecture consists of three sub-networks referred to as the perception, policy and behavior super-layers. The perception and behavior super-layers force an abstraction of visual and motor data trained with synthetic and simulated training samples, respectively. The policy super-layer is a small subnetwork with fewer parameters that maps data in-between the abstracted manifolds. It is trained for each task using methods for policy search reinforcement learning. We demonstrate the suitability of the proposed architecture and learning framework by training predictive policies for skilled object grasping and ball throwing on a PR2 robot. The effectiveness of the method is illustrated by the fact that these tasks are trained using only about 180 real robot attempts with qualitative terminal rewards.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, methodology): “In [13], the authors use a variational autoencoder trained on simulated data to encode trajectories as a lowdimensional latent code.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “a representation is found it allows for interpretable latent code manipulation, which is desirable in a variety of applications; recently, for example, in reinforcement learning [39, 18, 11, 41, 34].”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Deep neural network models have been used widely in a range of robotics applications [12, 13, 14, 15, 16, 17].”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “Reinforcement learning offers a promising avenue for tackling this problem, but current work on reinforcement learning tackles the problem of mastering individual skills, such as hitting a ball [1], opening a door [2, 3], or throwing [4].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “Prior work has demonstrated model-free RL in the real world with simulated pre-training [12] and parallelized data collection [13] for lower-dimensional whole-arm manipulation tasks.”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, methodology): “The model is close to [1]–[3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]–[38].”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “More recently, learning-based systems for robotic throwing [2, 14, 18, 11] have also been proposed, which ignore low-level dynamics and directly optimize for task-level success signals (e.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Model-free deep RL has been effective on short-horizon skills like object grasping [21, 20, 22], pushing and throwing [19, 23, 24], and multi-task learning [23, 25] from images.”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “This can be through human orchestrated resets [8, 13, 12, 4, 16], which requires high frequency human intervention in robot training.”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “Prior works have aimed to learn representations for control through auto-encoding [27, 48, 15, 16, 34], pretrained supervised features [40], spatial structure [15, 16, 23], and viewpoint invariance [41].”
Composable Deep Reinforcement Learning for Robotic Manipulation (2018, background): “[13] considers vision-based manipulation by training perception and behavior networks in simulation and learns only a low-dimensional intermediate layer with real-world interaction.”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…policies either from pixel inputs end-to-end (Haarnoja et al., 2018; Gelada et al., 2019; Singh et al., 2019; Kostrikov et al., 2020; Laskin et al., 2020; Han et al., 2020) or on top of unsupervised visual representations Lange and Riedmiller (2010); Ghadirzadeh et al. (2017); Nair et al. (2018).”
GPLAC: Generalizing Vision-Based Robotic Skills Using Weakly Labeled Images (2017, background): “However, this systems-level solution to robotic perception does not make use of the close cou-1 pling between perception and control that can be exploited by end-to-end methods to improve accuracy, dexterity, and robustness [27, 9, 52, 11, 39].”
Trends and challenges in robot manipulation (2019, background): “They are also capable of performing a variety of simple manipulation actions such as throwing ( 58 ), sliding ( 59 ), poking ( 60 ), pivoting ( 61 ), and pushing ( 62 ).”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “On-policy policy gradient algorithms have been used to learn tasks such as peg insertion (Lee et al. 2019), targeted throwing Ghadirzadeh et al. (2017), and dexterous manipulation (Zhu et al. 2019) directly on real-world hardware, and can be further accelerated with example demonstrations (Zhu et…”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “Recently, reinforcement learning has shown promise in completing complex tasks from pixels (Ghadirzadeh et al., 2017; Levine et al., 2015; Kalashnikov et al., 2018; Lange et al., 2012; OpenAI et al., 2018; Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including…”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “Another line of work (Ghadirzadeh et al., 2017; H¨am¨al¨ainen et al., 2019; Ghadirzadeh et al., 2020) explores using generative models for RL, using a variational autoencoder (Kingma & Welling, 2014) to model entire trajectories in an observation-independent manner, and then learning an open-loop,…”
Learning Visual Affordances for Robotic Manipulation (2019, background): “More recently, learning-based systems for robotic throwing [6, 45, 57, 30] have also been proposed, which ignore low-level dynamics and directly optimize for task-level success signals (e.”