Data-Efficient Generalization of Robot Skills with Contextual Policy Search

By Andras Gabor Kupcsik, Marc Peter Deisenroth, Jan Peters, Gerhard Neumann. 2013

135 citations recorded by the source corpus.

In robotics, controllers make the robot solve a task within a specific context. The context can describe the objectives of the robot or physical properties of the environment and is always specified before task execution. To generalize the controller to multiple contexts, we follow a hierarchical approach for policy learning: A lower-level policy controls the robot for a given context and an upper-level policy generalizes among contexts. Current approaches for learning such upper-level policies are based on model-free policy search, which require an excessive number of interactions of the robot with its environment. More data-efficient policy search approaches are model based but, thus far, without the capability of learning hierarchical policies. We propose a new model-based policy search approach that can also learn contextual upper-level policies. Our approach is based on learning probabilistic forward models for long-term predictions. Using these predictions, we use information-theoretic insights to improve the upper-level policy. Our method achieves a substantial improvement in learning speed compared to existing methods on simulated and real robotic tasks.

Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “successfully used in MBRL [8], experimental results [52] suggest that particle approaches can be competitive both computationally and in terms of accuracy, without making strong assumptions about the distribution used.”
One-Shot Visual Imitation Learning via Meta-Learning (2017, methodology): “Unlike prior methods that take the task identity [1, 2, 3, 4] or a demonstration [5] as the input into a contextual policy, our approach learns a parameterized policy that can be adapted to different tasks through gradient updates, effectively learning to imitation learn.”
Model-Free Preference-Based Reinforcement Learning (2016, methodology): “The policy update of actor critic REPS is inspired by the episodic REPS algorithm (Kupcsik et al. 2013).”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “There is also substantial prior work on learning reward or goal-conditional policies that directly produce actions [5, 6, 15, 35], but these rely either on a pre-trained world model, a dataset of goal and optimal policy parameters, or policy gradients for learning.”
Learning to Learn: Meta-Critic Networks for Sample Efficient Learning (2017, background): “Parametrised Tasks: In learning for control, the contextual policy or parametrised task setting considers actors that are parametrised by some metadata describing the goal they should achieve (e.g., the goal state of a mobile agent or robot arm to reach) [7, 19, 29].”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “, 1999) combines low-level option policies with a top-level policy that invokes individual options, whereas contextual policies (Kupcsik et al., 2013; Schaul et al., 2015; Heess et al., 2016) generalize options to continuous goals.”
Hindsight policy gradients (2017, background): “Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012; Kupcsik et ”
Model-Based Policy Gradients with Parameter-Based Exploration by Least-Squares Conditional Density Estimation (2013, unknown): “Combining (Kupcsik et al., 2013) or switching (Deisenroth et al.”
Tensor Based Knowledge Transfer Across Skill Categories for Robot Control (2017, background): “For example throwing objects of different weights [ Stulp et al. , 2014 ] , throwing to different target locations [ Kupcsik et al. , 2013 ] , or going via different waypoints to avoid various obstacles [ Stulp et al. , 2013 ] .”
Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control (2019, background): “[4] aims to learn a mapping from a given task parameter vector to a policy [4, 5, 6, 9, 16, 17, 21, 22, 24].”
Non-parametric Policy Search with Limited Information Loss (2017, methodology): “Earlier approaches based on the REPS formulation (Lioutikov et al., 2014; Kupcsik et al., 2013; Peters et al., 2010; Daniel et al., 2016) have shown to be successful on a variety of problems.”
A Survey on Policy Search for Robotics (2013, background): “However, the combination of model-free policy search with learned models seems to be a promising approach, a recent example is given in [42].”
Automatic Goal Generation for Reinforcement Learning Agents (2017, result): “Similar to previous work (Schaul et al., 2015; Kupcsik et al., 2013; Fabisch & Metzen, 2014; Deisenroth et al., 2014) we need a continuous goal-space representation such that a goal-conditioned policy can efficiently generalize over the goals.”
Information-Theoretic Motor Skill Learning (2013, methodology): “We further improved the learning speed of the contextual REPS algorithm by endowing our algorithm with learned Gaussian Process (Rasmussen and Williams 2006; Deisenroth and Rasmussen 2011) forward models (Kupcsik et al. 2013).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…learning into two categories: methods that seek to learn both the low-level and high-level policies through active interaction with an environment (Kupcsik et al., 2013; Heess et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Haarnoja et al., 2018a; Nachum et al., 2018; Chandak et al.,…”
Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences (2018, methodology): “Information-theoretic approaches like REPS attempt to stay close to the data while optimizing the policy.”