Black-box data-efficient policy search for robotics

By Konstantinos Chatzilygeroudis, Roberto Rama, Rituraj Kaushik, Dorian Goepp, Vassilis Vassiliades, Jean-Baptiste Mouret. 2017

115 citations recorded by the source corpus.

The most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynamical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot).

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “Indeed, model-free algorithms have previously been successfully used with particle trajectories from the same types of models as used in PILCO (Kupcsik et al., 2014), (Chatzilygeroudis et al., 2017).”
Data-efficient neuroevolution with kernel-based surrogate models (2018, methodology): “ad, we use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which has been proven as effective at optimizing Gaussian process model parameters as gradient-based methods in other contexts [5]. In addition to the kernel specific parametersℓ and ηthe mean (µ) and signal noise (σ) are also ”
Fast Model Identification via Physics Engines for Data-Efficient Policy Search (2017, methodology): “Replacing gradient-based optimization with a parallel, black-box algorithm (Covariance Matrix Adaptation (CMA)) [Hansen, 2006], Black-DROPS can be as data-efﬁcient as PILCO without imposing constraints on the reward function or the policy [Chatzilygeroudis et al. , 2017].”
Synthesizing Neural Network Controllers with Probabilistic Model-Based Reinforcement Learning (2018, methodology): “Black-DROPS [21] has recently shown promising performance competitive with the gradient-based PILCO [1] for training GP and NN policies using GP dynamics models.”
Reset-free Trial-and-Error Learning for Robot Damage Recovery (2016, methodology): “It is important to highlight that RTE is not a policy search method, like PILCO [17] or Black-DROPS [18]: RTE uses an approximate planner (MCTS) to derive a policy given the current model which, in turn, allows the robot to collect samples from the environment, refine the model, and thereby improve ”