Model-based contextual policy search for data-efficient generalization of robot skills
By Andras Gabor Kupcsik, Marc Peter Deisenroth, Jan Peters, Ai Poh Loh, Prahlad Vadakkepat, Gerhard Neumann. 2017
80 citations recorded by the source corpus.
In robotics, lower-level controllers are typically used to make the robot solve a specific task in a fixed context. For example, the lower-level controller can encode a hitting movement while the context defines the target coordinates to hit. However, in many learning problems the context may change between task executions. To adapt the policy to a new context, we utilize a hierarchical approach by learning an upper-level policy that generalizes the lower-level controllers to new contexts. A common approach to learn such upper-level policies is to use policy search. However, the majority of current contextual policy search approaches are model-free and require a high number of interactions with the robot and its environment. Model-based approaches are known to significantly reduce the amount of robot experiments, however, current model-based techniques cannot be applied straightforwardly to the problem of learning contextual upper-level policies. They rely on specific parametrizations of the policy and the reward function, which are often unrealistic in the contextual policy search formulation. In this paper, we propose a novel model-based contextual policy search algorithm that is able to generalize lower-level controllers, and is data-efficient. Our approach is based on learned probabilistic forward models and information theoretic policy search. Unlike current algorithms, our method does not require any assumption on the parametrization of the policy or the reward function. We show on complex simulated robotic tasks and in a real robot experiment that the proposed learning framework speeds up the learning process by up to two orders of magnitude in comparison to existing methods, while learning high quality policies.
Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jan Peters
- Sergey Levine
- Gerhard Neumann
- Pieter Abbeel
- Carl E. Rasmussen
- Marc Peter Deisenroth
- Jean-Baptiste Mouret
- J. Urgen Schmidhuber
In Reinforcement Learning
- Gerhard Neumann
- Jan Peters
- Sergey Levine
- Pieter Abbeel
- Marc Peter Deisenroth
- Carl E. Rasmussen
- Christian Wirth
- Johannes Fürnkranz
In Robotics & Control
- Gerhard Neumann
- Jan Peters
- Christian Wirth
- Johannes Fürnkranz
- Jean-Baptiste Mouret
- Vassilis Vassiliades
- Konstantinos Chatzilygeroudis
- Riad Akrour
Citations in the corpus
- PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “Indeed, model-free algorithms have previously been successfully used with particle trajectories from the same types of models as used in PILCO (Kupcsik et al., 2014), (Chatzilygeroudis et al.”
- Learning Robust Policies for Object Manipulation with Robot Swarms (2018, methodology): “Inspired by the episodic REPS algorithm [18], AC-REPS solves this problem by using information-theoretic constraints.”
- Black-box data-efficient policy search for robotics (2017, methodology): “Gradient-free methods, such as the Model-Based Relative Entropy PS (M-REPS) [29] and the Model-Based Guided PS (M-GPS) [27], do not have these requirements.”
- A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “For example, it can learn to adapt the robot’s trajectory if we need to throw a ball for a larger distance (Kupcsik et al., 2014).”